Communication is the most critical thing in the world which ties the whole world together. There are various mediums of communication: voice, video, and text. Each medium brings its own benefits based on the context. Technology has made significant progress in providing interfaces using these mediums. These mediums are used for human-to-machine, human-to-human, AI-generated, and in some basic form machine-to-machine communication (I bet machine-to-machine is going to be very sophisticated in the short term future as this would bring a lot more use cases)
AI, ML, and NLP have made huge progress in recent times to automate these conversations or communication. The basic theory is to have a 24*7 working system (For example automated customer support) and auto-scale to handle the load on the system (Example: A surge in support calls because some bugs crept in at a new release). Such automation systems save cost and automated systems provide higher quality as there is no human error involved.
As these automated communications grow in technologies and use cases, there is a flip side to this too. Now attackers have access to AI tools and scalable systems in their hands to attack such systems. As these systems are being incorporated into the critical components of our livelihood, they are more vulnerable to cybersecurity attacks and bring critical harm to the system. We already have instances of AI-generated fake Twitter messages, news, and videos causing unrest in society. Automated attacks bring scale and that has so much harmful impact on society.
What is Conversational AI?
Conversational artificial intelligence (AI) is the application of natural language processing (NLP) and artificial intelligence (AI) to make machines converse like humans. Conversational AI aims to build systems that mimic real-world human communication by understanding, interpreting, and reacting to user inputs.
Chatbots, which employ natural language processing (NLP) to interpret user input and conduct conversations, are among the most popular uses of conversational AI. Chatbots for customer service, voice assistants, and virtual assistants are some more uses.
The objective of Conversational AI
- Conversational systems are one of the most advanced developments of AI that actively help and impact our lives. One of the key practical uses of conversational systems in today’s world is to automate human work to assist in tasks that we perform daily. These systems work efficiently throughout the day and are capable of handling high loads of data. Conversational system usage has grown multifold during pandemic days and continues to be upward like a hockey stick.
- The objective of this research paper is to identify vulnerabilities of conversational systems. These attacks are new and there is almost nil or zero research done to date. Also, the paper proposes some of the techniques to defend against these attacks. Experimentation was carried out to simulate some of the plausible attacks and proposed algorithms were implemented to see the effectiveness of the algorithm against these attacks.
Key components of conversational AI
Natural Language Processing (NLP): The field of artificial intelligence that focuses on natural language interaction between computers and people is called natural language processing, or NLP. In order to derive meaning from speech or text, one must comprehend language semantics, syntax, and context.
Speech Recognition: With the use of this technology, machines can now translate spoken words into written text. Building conversational interfaces that can comprehend spoken language and react to it requires it.
Machine Learning: To continuously enhance its capabilities, conversational AI frequently uses machine learning techniques. These algorithms have the capacity to learn from data, adjust to user behaviour, and improve the system’s comprehension of responses and production of more contextually appropriate ones.
Dialog Management: The process of planning and arranging a conversation is known as “dialogue management.” It entails monitoring context, controlling turn-taking, and making sure that the conversation flows naturally.
Intent Recognition: In order to respond appropriately, it is necessary to ascertain the user’s intent. Determining the user’s intent or request from their input is known as intent recognition.
Context Awareness: In order to better comprehend user inquiries, conversational AI systems strive to preserve context throughout a dialogue. Recalling prior exchanges, user preferences, and other pertinent data is necessary for this.
Conversational AI Real-World Cases
Many industries have found use for conversational AI, which improves user experiences, offers customer support, and streamlines procedures. Here are a few examples of conversational AI in action:
Chatbots and Virtual Assistants for Customer Service:
- Amazon Alexa: Created by Amazon, Alexa is a voice-activated virtual assistant. It functions with gadgets similar to the Amazon Echo, enabling users to give voice commands for controlling smart home appliances, playing music, getting answers to questions, and more.
- Google Duplex: An AI system created for natural language communication is called Google Duplex. It has sophisticated conversational abilities and can schedule appointments, book restaurants, and carry out other tasks over the phone.
- Banking Chatbots: A lot of banks use chatbots to help consumers with simple questions, account details, and past transactions. These chatbots are able to respond quickly and guide users through different banking services.
Uses in Healthcare:
- Chatbots for Health Information: In the healthcare industry, chatbots are used to help users make appointments, share information about symptoms, and provide preliminary medical advice. By giving users a conversational interface to express their emotions, they can also help with mental health support.
- Chatbots for Medication Management: Some apps use conversational AI to assist users in taking their medications on time. These chatbots can provide users information about possible side effects, answer questions regarding dosage, and remind users to take their medications.
Retail and E-commerce:
- Virtual Shopping Assistants: To build virtual shopping assistants that assist users in finding products, comparing prices, and making decisions about what to buy, conversational AI is used. These helpers are capable of comprehending natural language inquiries and offering tailored advice.
- Chatbots for order tracking: Online retailers employ chatbots to assist consumers with order tracking, delivery-related inquiries, and real-time shipment status updates.
Education and Training:
- Language Learning Chatbots: To give users a conversational practise partner, conversational AI is employed in language learning applications. Through conversational exchanges, these chatbots assist language learners in strengthening their language abilities.
- Assistants for Training and Onboarding: Chatbots are utilised in business environments to help with employee onboarding and training. They can help new hires get started in their roles, answer questions, and offer information.
Tourism and Hospitality:
- Reservation and Booking Assistance: Chatbots help consumers make reservations for hotels, rental cars, and flights. Using natural language interactions, they can comprehend user preferences, offer options, and streamline the booking process.
- Customer Support in Hospitality: In the hospitality industry, chatbots are used by hotels and travel agencies to instantly respond to client inquiries about bookings, features, and nearby attractions.
Conversational Systems Security Risks
Conversation systems are vulnerable to many attacks specifically when automated as they lack the identification of human vs machine-generated conversations. Also, these systems are built on AI/ML and thus inherit the higher security vulnerabilities of AI systems. Natural language processing is used by conversational systems as an interface layer that enables efficient interactions with end-users adding an extra threat vector to the existing ML system threats.
Recent advancements with NLP have been a few years in the making, starting in 2018 with the launch of two massive deep learning models: GPT (Generative Pre-Training) by Open AI, and BERT (Bidirectional Encoder Representations from Transformers) for language understanding, including BERT-Base and BERT-Large by Google. Unlike previous NLP models, BERT is an open-source deeply bidirectional, and unsupervised language representation, which is pre-trained solely using a plain text corpus. Since then we have seen the development of other deep learning massive language models: GPT-2, RoBERT, ESIM+GloVe, and now GPT-3.
These tools make it so easy to generate human-generated look-like text and that provides opportunities for attackers to fool or malfunction the conversational AI systems. Also, systems this conversational system automates a lot more customer interactions to save human energy for much more complex natural work. An example of such a task could be a customer asking the bank about its opening time.
The author of this paper did research by talking to various banking professionals all over the world and figured out that trivial queries make up 85% of the customer’s queries received every day.
The following are the most common security attacks on conversational systems.
1. Adversarial Attacks/Filter Evasion:
Adversarial attacks/Filter evasion also called input attacks are the most common type of attack a conversational AI/ML system faces. Attackers craft an attack based on the information available to them and exploit the weakness in the ML/NLP models. The attackers manipulate the ML system by incorporating malicious inputs causing the system to make false predictions.
Example: Crafting text in a way to bypass profanity filters to publish news that local government agencies restrict.
There was an experiment done with Microsoft Text Analytics API which provides profanity filters. Masked output from Microsoft API when there was no adversarial text. This is a s**t product I would have purchased to date. F****g vendor and f*****g seller. However, when input like this was sent. This is a shit1 product I would have purchased to date. Fuccing vendor and fu*cing seller. There was no adversarial masking was executed.
Google Sentiment Analysis API
- This is the worst movie I have seen ever. Negative (95% confidence)
- This is the worst movie I have seen ever. Positive (95% confidence)
There are plenty of open-source repositories available for attacking the text classifier. A lot many functionalities in NLP NLP-based system depend on the classifier and if the classifier is tricked to classify as per the attacker’s need, it could impose a serious threat.
Some examples of such repositories are
- https://github.com/jind11/TextFooler
- https://github.com/QData/TextAttack
- https://paperswithcode.com/task/adversarial-text
These attacks would have a severe impact when human and machine interpretation of the text is different. So as a human eye, I would classify differently from the machine and that’s why these attacks are named adversarial. The basic idea of such an attack is to bypass human eyes.
2. Data/Analytics Poisoning:
As a conversational system is built on top of AI/ML and dependent on data, corrupting it can result in system malfunctions. AI systems work by learning the task from the data which is obtained from various sources. Poisoning the data will directly result in poisoning the conversational system thus resulting in making wrong decisions.
Example: Fake queries or posting fake recommendations of a product to make it one of the most popular products. The pandemic has a massive impact on our day-to-day life. And life has become more online than offline. Many online retailers promote products by using customer recommendations and we as buyers pay a lot of attention to recommendations. Now imagine someone automating such recommendations to send a text to online retailers and impacting the whole product ratings. Continuing the same think about when adversarial text is also added to recommendations. Now humans interpret the same text just opposite to the text interpreted by machines.
Let’s understand this with an example: If an attacker can misclassify product recommendations, it would have a machine-direct impact on the revenue. Machines would rate a product higher based on the classified recommendations, but humans would see the product different way.
3. Fake Requests/Transactions using Bots/AI Bots:
These attacks are becoming very easy to execute as the AI system is getting so advanced. Attackers can very easily use cloud infrastructure to simulate fake requests and transactions using AI which mimics human behavior.
Example: Many organizations have moved to automated support now. This trend has gone multifold during the pandemic as there was no one to in-office physically present in the office to fulfill customer’s requests. Expectations or prediction is to see this going up and up.
Advanced NLP has given easiness and sophistication to create bots that could pretty much respond like humans to hum’s queries. But attackers could exactly use the same bots in other directions as well. AI bots can easily generate fake sales inquiries which would generate fake sales leads. This could be very harmful if the system cannot isolate fake leads from genuine ones. This has huge potential to ignore genuine customers and end up fulfilling the request of a bot. This has the potential of creating a bad reputation and as well loss of revenue.
Example: Keep sending product inquiries or dummy complaints or buying orders using a bot in bulk. This would make sure that genuine requests too are lost and it means a direct loss of revenue.
4. Social Engineering Attacks:
As one of the most popular social engineering attack types, Phishing scams are email and text message campaigns aimed at creating a sense of urgency, curiosity, or fear in victims. It then prods them into revealing sensitive information, clicking on links to malicious websites, or opening attachments that contain malware.
Example: Email sent to users of an online service that alerts them of a policy violation requiring immediate action on their part, such as a required password change.
5. Intelligent DDOS Attacks:
Traditional web applications were facing DDOS attacks which were volume-based like attackers sent a high volume of HTTP requests Also they did not expose interfaces that were automating the response of a human. Such DDOS attacks are not Denial of services from software systems as such. These Denial of services are Denial of services provided by humans like healthcare professionals. Model conversational AI systems work on behalf of humans (like customer support systems or automated appointments) and that makes them vulnerable to Intelligent DDOS attacks without using high numbers of HTTP requests.
Example: Attackers can use AI bots to book most of the time slots in healthcare service provider systems. This would result in the denial of services to the patient who is in greater need of the slot.
AI Bots can ask queries to chatbots which are expensive in terms of execution or escalate all the calls to humans, defeating the purpose of chatbot deployments could result in low ROI. If the system is built on technologies which is costlier like elastic search.
6. Generate Unanswered Queries:
The conversational AI system works on the principle of improving all the time based on the feedback. This feedback is the result of failed requests from customers. This is a semi-automated process as of now. All the unanswered queries are redirected to a centralized place. Though some automation could be applied here a mostly human goes through the unanswered queries and the system is trained to answer those queries. This does to the model and a new system is deployed. This is an iterative process.
Example: Attackers can use AI bots to ask those queries which could create a huge volume of such unanswered queries. Now it would always be a mix of genuine and bot-generated queries that are not answered. This results in spending time on queries that are not genuine, and some genuine queries are missed out.
7. Route Support Request to Human:
The role of chatbots is going to be bigger and better. With the emerging chatbot trends and market outlook, businesses must adopt innovative ways to deliver continuous customer engagement. As per Gartner, “Artificial Intelligence (AI) will be a mainstream customer experience investment in the next couple of years”. 47% of organizations will use chatbots for customer care and 40% will deploy virtual assistants.
AI has been revamping the ways of communication ways for businesses both with customers and internally. AI is vital for enabling machine learning and the flexible interpretation of automated business communications. Going further, chatbots are predicted to move from simple user-based queries to more advanced predictive analytics-based real-time conversations.
Example: Most of the chatbots are designed to handle L1 support and as things get complex, there is always an option to route the request to humans. Now imagine a scenario where an AI bot continues to route the requests to humans for the next level of support. This would defeat the whole purpose of deploying chatbots. Also, since human support staff is precious (and of course less), this has the potential to bring down the support system.
8. DDOS via Various Conversation Channels:
Conversation’s advancement has also added various physical mediums for interacting with end-users. This provides a lot of flexibility to the end-user but also opens up the gates for attackers to find the flexibilityhumansweakness in the system by using multiple physical channels.
Example: Attackers could use an AI bot to send similar requests from multiple channels. Typically, the eventual processing of the requests is done by the same server. Now such channels add parallelism for the attacker, and it could very easily choke the processing server.
Frequently Asked Questions(FAQs)
Q. 1 What is Conversational AI?
Conversational artificial intelligence (AI) is the application of artificial intelligence (AI) technologies, like machine learning and natural language processing (NLP), to make machines converse like human beings. It attempts to mimic real-world communication by comprehending, interpreting, and reacting to user inputs.
Q. 2 How does Conversational AI work?
Conversational AI systems employ natural language processing (NLP) to comprehend and analyse spoken or written user input. By using data and user interactions to learn, machine learning algorithms are essential for gradually enhancing system performance. To facilitate coherent and contextually relevant conversations, three essential elements are needed: context awareness, intent recognition, and dialogue management.
Q. 3 What are some real-world applications of Conversational AI?
Applications for conversational AI can be found in virtual assistants (like Google Duplex and Amazon Alexa), chatbots for customer service, healthcare information bots, virtual shopping assistants, applications for language learning, and much more. It improves user experiences, expedites procedures, and offers tailored support across multiple domains.
Q. 4 How is Conversational AI evolving in the future?
Conversational AI’s future promises better language generation, enhanced personalization, more seamless integration into everyday life, and a better understanding of context. It is anticipated that conversational AI will advance in sophistication and intuitiveness as technology develops.