AI Conversational System – Attack Surface Areas and Effective Defense Techniques

Background:

Communication is the most critical thing in the world which ties the whole world together. There are various mediums of communication: voice, video, and text. Each medium brings its own benefits based on the context. Technology has made significant progress to provide interfaces using these mediums. These mediums are used for human to machine, human to human, AI-generated, and in some basic form machine-to-machine communication (I bet machine to machine is going to be very sophisticated in short term future as this would bring a lot more use cases)  

AI, ML or NLP has made huge progress in recent times to automate these conversations or communication. The basic theory is to have a 24*7 working system (For example like automated customer support) and auto-scale to handle the load on the system (Example: Surge in support calls because some bugs crept in at a new release). Such automation systems save cost and automated systems provide higher quality as there is no human error involved.

As these automated communications grow in technologies and use cases, there is a flip side to this too. Now attackers have access to AI tools and scalable systems in their hands to attack such systems. As these systems are being incorporated into the critical components of our livelihood, these are more vulnerable to cybersecurity attacks and bring critical harm to the system.  We already have instances of AI-generated fake Twitter messages, news, and video causing unrest in society. Automated attacks bring scale and that has so much harmful impact on society.

Objective:

Conversational systems are one of the most advanced developments of AI that actively help and impact our lives. One of the key practical usage of conversational systems in today’s world is to automate human work to assist in tasks that we perform daily. These systems work efficiently throughout the day and are capable of handling high loads of data. Conversational system usage has grown multifold during pandemic days and continues to be upward like a hockey stick.

The objective of this research paper is to identify vulnerabilities of conversational systems. These attacks are new and there is almost nil or zero research done to date. Also, the paper proposes some of the techniques to defend against these attacks. Experimentation was carried out to simulate some of the plausible attacks and proposed algorithms were implemented to see the effectiveness of the algorithm against these attacks.

Conversational Systems Security Risks:

Conversation systems are vulnerable to many attacks specifically when automated as they lack the identification of human vs machine-generated conversations. Also, these systems are built on AI/ML and thus inherit the higher security vulnerabilities of AI systems.  Natural language processing is used by conversational systems as an interface layer that enables efficient interactions with end-users adding an extra threat vector on the existing ML system threats.  

Recent advancements with NLP have been a few years in the making, starting in 2018 with the launch of two massive deep learning models: GPT (Generative Pre-Training) by Open AI, and BERT (Bidirectional Encoder Representations from Transformers) for language understanding, including BERT-Base and BERT-Large by Google. Unlike previous NLP models, BERT is an open-source and deeply bidirectional and unsupervised language representation, which is pretrained solely using a plain text corpus. Since then we have seen the development of other deep learning massive language models: GPT-2, RoBERT, ESIM+GloVe, and now GPT-3.

These tools make it so easy to generate human-generated look like text and that provides opportunities for attackers to fool or malfunction the conversational AI systems. Also, systems this conversational system automates a lot more customer interactions to save human energy for much more complex nature work. An example of such a task could be a customer asking the bank about its opening time.  

The author of this paper did research by talking to various banking professionals all over the world and figured out trivial queries make 85% of the customer’s queries received every day.

The following are the most common security attacks on conversational systems.

1.   Adversarial Attacks/Filter Evasion:

Adversarial attacks/Filter evasion also called input attacks are the most common type of attack a conversational AI/ML system faces. Attackers craft an attack based on the information available to them and exploit the weakness in the ML/NLP models. The attackers manipulate the ML system by incorporating malicious inputs causing the system to make false predictions.  

Example: Crafting text in a way to bypass profanity filters to publish news that is restricted by local government agencies.

There was an experiment done with Microsoft Text Analytics API which provides profanity filters. Masked output from Microsoft API when there was no adversarial text. This is absolutely s**t product I would have purchased to date. Absolutely f****g vendor and f*****g seller. However, when input like this was sent. This is absolutely shit1 product I would have purchased to date. Absolutely fuccing vendor and fu*cing seller. There was no adversarial masking was executed.

Google Sentiment Analysis API

• This is the worst movies I have seen ever. Negative (95% confidence)  
• This is the wo1rst movies I have seen ever. Positive (95% confidence)

There are plenty of open-source repositories available for attacking the text classifier. Lot many functionalities in NLP based system depends on the classifier and if the classifier is tricked to classify as per the attacker’s need, it could impose a serious threat.

Some of the examples of such repositories are  

• https://github.com/jind11/TextFooler
• https://github.com/QData/TextAttack
• https://paperswithcode.com/task/adversarial-text

These attacks would have a severe impact when human and machine interpretation of the text is different. So as a human eye I would classify different from the machine and that’s why these attacks are named adversarial. The basic idea of such an attack is to bypass human eyes.

2. Data/Analytics Poisoning:  

As a conversational system is built on top of AI/ML and dependent on data, corrupting it can result in system malfunctions. AI systems work by learning the task from the data which is obtained from various sources. Poisoning the data will directly result in poisoning the conversational system thus resulting in making wrong decisions.

Example:  Fake queries or posting fake recommendations of a product to make it one of the most popular products. Pandemic has a massive impact on our day-to-day life. And life has become more online than offline. Many online retailers promote products by using customer recommendations and as well we as buyers pay a lot of attention to recommendations. Now imagine someone automating such recommendations to send a text to online retailers and impacting the whole product ratings. Continuing the same think about when adversarial text is also added to recommendations. Now humans interpret the same text just opposite to the text interpreted by machines.

Let’s understand this with an example: If an attacker can misclassify product recommendations, it would have a machine-direct impact on the revenue. Machine would rate a product higher based on the classified recommendations, but humans would see the product different way.  

3. Fake Requests/Transactions using Bots/AI Bots:

These attacks are becoming very easy to execute as the AI system is getting so advanced. Attackers can very easily use cloud infrastructure to simulate fake requests and transactions using AI which mimics human behavior.

Example: Many organizations have moved to automated support now. This trend has gone multifold during the pandemic as there was no one to in-office physically present in the office to fulfill customer’s requests. Expectations or prediction is to see this going up and up .

Advanced of NLP has given easiness and sophistication to create bots that could pretty much respond like humans to hum’s queries. But attackers could exactly use the same bots in other directions as well. AI bots can easily generate fake sales inquiries the which would generate fake sales leads. This could be very harmful if the system cannot isolate fake leads from genuine ones. This has huge potential to ignore genuine customers and end up fulfilling request of a bot. This has the potential of creating a  bad reputation and as well loss of revenue.

Example: Keep sending product inquiries or dummy complaints or buying orders using a bot in bulk. This would make sure that genuine requests too are lost and it means a direct loss of revenue.

4. Social Engineering Attacks:

As one of the most popular social engineering attack types, Phishing scams are email and text message campaigns aimed at creating a sense of urgency, curiosity, or fear in victims.  It then prods them into revealing sensitive information, clicking on links to malicious websites, or opening attachments that contain malware.

Example: Email sent to users of an online service that alerts them of a policy violation requiring immediate action on their part, such as a required password change.

5. Intelligent DDOS Attacks:

Traditional web applications were facing DDOS attacks which were volume-based like attackers sent a high volume of http requests Also they did not expose interfaces that were automating the response of a human. Such DDOS attacks are not Denial of services from software systems as such. These Denial of services are Denial of services provided by humans like healthcare professionals. Model conversational AI systems work on behalf of humans (like customer support systems or automated appointments) and that makes them vulnerable to Intelligent DDOS attacks without using high numbers of http requests.

Example: Attackers can use AI bots to book most of the time slots in healthcare service provider systems. This would result in denial of services to the patient who is in greater need of the slot.

AI Bots can ask queries to chatbots which are expensive in terms of execution or escalate all the calls to humans, defeating the purpose of chatbot deployments could result as well in low ROI. If the system is built on technologies which is costlier like elastic search.

6. Generate Unanswered Queries:

Conversational AI system works on the principle of improving all the time based on the feedback. This feedback is the result of failed requests from customers. This is semi-automated process as of now. All the un-answered queries are redirected to a centralized place. Though there is some automation that could be applied here a mostly human goes through the unanswered queries and the system is trained to answer those queries. This does to the model and a new system is deployed. This is an iterative process.

Example: Attackers can use AI bots to ask those queries which could create a huge volume of such unanswered queries. Now it would always be a mix of genuine and bot-generated queries which are not answered. This results in spending time on queries that are not genuine, and some genuine queries to are missed out.

7. Route Support Request to Human:

 The role of chatbots is going to be bigger and better. With the emerging chatbots trends and market outlook, it is crucial for businesses to adopt innovative ways to deliver continuous customer engagement. As per Gartner, “Artificial Intelligence (AI) will be a mainstream customer experience investment in the next couple of years”. 47% of organizations will use chatbots for customer care and 40% will deploy virtual assistants.

AI has been revamping the ways of communication ways for businesses both with customers and internally. AI is vital for enabling machine learning and the flexible interpretation of automated business communications. Going further, chatbots are predicted to move from simple user-based queries to more advanced predictive analytics-based real-time conversations.  

Example: Most of the chatbots are designed to handle L1 support and as things get complex, there is always an option to route the request to humans. Now imagine a scenario where an AI bot continues to route the requests to humans for the next level of support.  This would defeat the whole purpose of deploying chatbots. Also, since human support staff is precious (and of course less), this has the potential of bringing down the support system.

8. DDOS via Various Conversation Channels:

Conversation’s advancement has also added various physical mediums for interacting with end-users. This provides a lot of flexibly to the end-user but also opens up the gates for attackers to find the flexibilityhumansweakness in the system by using multiple physical channels. 

Example: Attackers could use an AI bot to send similar requests from multiple channels. Typically, the eventual processing of the requests is done by same server. Now such channels add parallelism for the attacker, and it could very easily choke the processing server.