Intent Recognition using TensorFlow

Intent recognition is a method of natural language processing, which deals with determining intent of a given sentence, or in simple terms “what the sentence means”. It is commonly used in chatbots, virtual assistants, and other conversational AI systems to understand user requests and provide appropriate responses.

In this article, We will show a basic of how Intent Recognition can be made using TensorFlow

TensorFlow : TensorFlow is an open-source library for machine learning developed by Google. It provides various tools and functions to build and train neural networks efficiently, and a neural network is a computational model inspired by the functioning of the human brain, composed of interconnected artificial neurons that can learn and make predictions from input data.

To perform Intent Recognition using TensorFlow, we typically follow these steps:

Import Libraries:

We start by importing all necessary modules.

Python3

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import numpy as np
import random
import json

import warnings
warnings.filterwarnings('ignore')

We import ‘tensorflow' to access the core TensorFlow functionality, ‘Tokenizer' to preprocess text data by tokenizing it into sequences, and ‘pad_sequences' to pad the sequences to a fixed length.

Data Collection:

Then, we need a dataset consisting of labeled examples where each example contains a text input and its corresponding intent label. For example, we consider a dataset from kaggle, to view: Chatbots: Intent Recognition Dataset, it is in JSON format and perfect for intent recognition model.

Python3

with open('Intent Recognition/Intent.json', 'r') as f:
    data = json.load(f)

print(data.keys())
print(type(data['intents']))
print(len(data['intents']))
print(data['intents'][0].keys())
data['intents'][-1]

Output:

dict_keys(['intents'])
<class 'list'>
22
dict_keys(['intent', 'text', 'responses', 'extension', 'context', 'entityType', 'entities'])
{'intent': 'SelfAware',
 'text': ['Can you prove you are self-aware',
  'Can you prove you are self aware',
  'Can you prove you have a conscious',
  'Can you prove you are self-aware please',
  'Can you prove you are self aware please',
  'Can you prove you have a conscious please',
  'prove you have a conscious'],
 'responses': ['That is an interesting question, can you prove that you are?',
  'That is an difficult question, can you prove that you are?',
  'That depends, can you prove that you are?'],
 'extension': {'function': '', 'entities': False, 'responses': []},
 'context': {'in': '', 'out': '', 'clear': False},
 'entityType': 'NA',
 'entities': []}

We load data from JSON file like shown above.

Data Cleaning:

We can’t directly process this data. First, we have to clean it, this usually includes, removing punctuation marks or anything which might produce unwanted results. This is also done to reduce number of tokens, as we don’t want to tokenize everything.

Python3

def clean(line):
    cleaned_line = ''
    for char in line:
        if char.isalpha():
            cleaned_line += char
        else:
            cleaned_line += ' '
    cleaned_line = ' '.join(cleaned_line.split())
    return cleaned_line

We define a function to clean sentence provided to it, by removing punctuations. This can also be done by using Regular Expression module in python.

Data Preprocessing:

The next step is to preprocess the data to make it suitable for training a neural network. This usually involves tokenization, which means breaking down each input sentence into individual words or sub-words.

But before we do that, we need to prepare data in a format {intent : text data}

Python3

#list of intents
intents = []
unique_intents = []
#all text data to create a corpus
text_input= []
#dictionary mapping intent with appropriate response
response_for_intent = {}
for intent in data['intents']:
    #list of unique intents
    if intent['intent'] not in unique_intents:
        unique_intents.append(intent['intent'])
    for text in intent['text']:
        #cleaning is done before adding text to corpus
        text_input.append(clean(text))
        intents.append(intent['intent'])
    if intent['intent'] not in response_for_intent:
        response_for_intent[intent['intent']] = []
    for response in intent['responses']:
        response_for_intent[intent['intent']].append(response)

We have created a dataset which is ready to be preprocessed, we can see sample data below:

Python3

print("Intent :",intents[0])
print("Number of Intent:",len(intents))
print("Sample Input:", text_input[0])
print('Length of text_input:',len(text_input))
print("Sample Response: ", response_for_intent[intents[0]])

Output:

Intent : Greeting
Number of Intent: 143
Sample Input: Hi
Length of text_input: 143
Sample Response:  ['Hi human, please tell me your GeniSys user',
'Hello human, please tell me your GeniSys user',
 'Hola human, please tell me your GeniSys user']

Tokenization and Embedding

Now, our data is ready to be tokenized, with the help of inbuilt TensorFlow tokenizer, We can make both tokenization and embedding.

Python3

tokenizer = Tokenizer(filters='',oov_token='<unk>')
tokenizer.fit_on_texts(text_input)
sequences = tokenizer.texts_to_sequences(text_input)
padded_sequences = pad_sequences(sequences, padding='pre')
print('Shape of Input Sequence:',padded_sequences.shape)
padded_sequences[:5]

Output:

Shape of Input Sequence: (143, 9)
array([[ 0,  0,  0,  0,  0,  0,  0,  0, 52],
       [ 0,  0,  0,  0,  0,  0,  0, 52, 53],
       [ 0,  0,  0,  0,  0,  0,  0,  0, 68],
       [ 0,  0,  0,  0,  0,  0,  0,  0, 39],
       [ 0,  0,  0,  0,  0,  0,  0, 39, 53]], dtype=int32)

Feature Extraction:

Neural network cannot process sentences, so numerical representation of sentences have to be provided to it, this is done by doing Feature Extraction, for that we map all words with their indexes and create a matrix mapping it to its category (intent).

Python3

intent_to_index = {}
categorical_target = []
index = 0

for intent in intents:
    if intent not in intent_to_index:
        intent_to_index[intent] = index
        index += 1
    categorical_target.append(intent_to_index[intent])

num_classes = len(intent_to_index)
print('Number of Intents :',num_classes)

# Convert intent_to_index to index_to_intent
index_to_intent = {index: intent for intent, index in intent_to_index.items()}
index_to_intent

Output:

Number of Intents : 22
{0: 'Greeting',
 1: 'GreetingResponse',
 2: 'CourtesyGreeting',
 3: 'CourtesyGreetingResponse',
 4: 'CurrentHumanQuery',
 5: 'NameQuery',
 6: 'RealNameQuery',
 7: 'TimeQuery',
 8: 'Thanks',
 9: 'NotTalking2U',
 10: 'UnderstandQuery',
 11: 'Shutup',
 12: 'Swearing',
 13: 'GoodBye',
 14: 'CourtesyGoodBye',
 15: 'WhoAmI',
 16: 'Clever',
 17: 'Gossip',
 18: 'Jokes',
 19: 'PodBayDoor',
 20: 'PodBayDoorResponse',
 21: 'SelfAware'}

One-Hot Encoding

Apply One-Hot Encoding for categorical_target

Python3

categorical_vec = tf.keras.utils.to_categorical(categorical_target,
                                                num_classes=num_classes, dtype='int32')

print('Shape of Ca',categorical_vec.shape)
categorical_vec[:5]

Output:

Shape of Ca (143, 22)
array([[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
      dtype=int32)

Model Building:

First step of building a model is defining hyperparameters, in simple words they are settings which we predefine before training our model. They include parameters like no. of epochs, embedding dimensions, size of vocabulary, and target length. They can be adjusted accordingly, to increase performance of model.

Python3

epochs=100
embed_dim=300
lstm_num=50
output_dim=categorical_vec.shape[1]
input_dim=len(unique_intents)
print("Input Dimension :{},\nOutput Dimension :{}".format(input_dim,output_dim))

Output:

Input Dimension :22,
Output Dimension :22

As both input dimension and output dimension are same, we can proceed with building our model.

Now, we can define the architecture of our neural network using TensorFlow. A common model for intent recognition is the recurrent neural network (RNN) or its variant, the long short-term memory (LSTM) network. These networks can handle sequential data, such as sentences, effectively. We can also use pre-trained models like BERT or GPT to achieve better performance.

Here we are using a RNN, by using ‘Sequential’ model from TensorFlow’s Keras API. It consists of an embedding layer, an LSTM layer for sequence processing, and two dense layers for classification. You can see model summary below.

Python3

model = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(len(tokenizer.word_index) + 1, embed_dim),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(lstm_num, dropout=0.1)),
    tf.keras.layers.Dense(lstm_num, activation='relu'),
    tf.keras.layers.Dropout(0.4),
    tf.keras.layers.Dense(output_dim, activation='softmax')
])

optimizer = tf.keras.optimizers.Adam(lr=0.001)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()

Output:

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding (Embedding)       (None, None, 300)         35700     
                                                                 
 bidirectional (Bidirectiona  (None, 100)              140400    
 l)                                                              
                                                                 
 dense (Dense)               (None, 50)                5050      
                                                                 
 dropout (Dropout)           (None, 50)                0         
                                                                 
 dense_1 (Dense)             (None, 22)                1122      
                                                                 
=================================================================
Total params: 182,272
Trainable params: 182,272
Non-trainable params: 0
_________________________________________________________________

Training:

With the model architecture defined, we can start training the network using our labeled dataset. During training, the model adjusts its internal parameters to minimize the difference between its predicted intents and the true intents in the training data. This process involves forward propagation, where the model makes predictions, and backward propagation, where the model’s parameters are updated based on the prediction errors.

Python3

model.fit(padded_sequences, categorical_vec, epochs=epochs, verbose=0)

Output:

<keras.callbacks.History at 0x7f75fff7df10>

Here we train the model by fitting it to the padded sequences and its respective categorical sequence. The model adjusts its internal parameters to minimize the difference between its predicted intents and the true intents in the training data. The training is performed for a specified number of epochs, and might stop early if there is no improvement in it’s loss after 4 consecutive epochs.

Evaluate:

To check if our model works correctly, we give it unseen data along with labels and check if it works correctly or not. Thus, we give text inputs along with it’s intent and we test our model, this is known as evaluation of model. It is important step which helps predict accuracy of model, or if it is overfit or underfit.

Python3

test_text_inputs = ["Hello",
                    "my name is adam",
                    "how are you?",
                    "can you guess my name?",
                    "Do you get me","Adios"]

test_intents = ["Greeting",
                "GreetingResponse",
                "CourtesyGreeting",
                "CurrentHumanQuery",
                "UnderstandQuery",
                "GoodBye"]

test_sequences = tokenizer.texts_to_sequences(test_text_inputs)
test_padded_sequences = pad_sequences(test_sequences,  padding='pre')
test_labels = np.array([unique_intents.index(intent) for intent in test_intents])
test_labels = tf.keras.utils.to_categorical(test_labels, num_classes=num_classes)
loss, accuracy = model.evaluate(test_padded_sequences, test_labels)

Output:

1/1 [==============================] - 1s 572ms/step - loss: 0.4058 - accuracy: 1.0000

Predict:

We have finished training our model and can now use it to make predictions on new, unseen sentences. Given an input sentence, the model processes it using the learned parameters and produces a probability distribution over the possible intents. The intent with the highest probability is considered the predicted intent.

Python3

def response(sentence):
    sent_tokens = []
    # Split the input sentence into words
    words = sentence.split()
    # Convert words to their corresponding word indices
    for word in words:
        if word in tokenizer.word_index:
            sent_tokens.append(tokenizer.word_index[word])
        else:
            # Handle unknown words
            sent_tokens.append(tokenizer.word_index['<unk>'])
    sent_tokens = tf.expand_dims(sent_tokens, 0)
    #predict numerical category
    pred = model(sent_tokens)
    #category to intent
    pred_class = np.argmax(pred.numpy(), axis=1)
    # random response to that intent
    return random.choice(
        response_for_intent[index_to_intent[pred_class[0]]]), index_to_intent[pred_class[0]]

Chatbots: Intent Recognition

We have defined a function response to predict intent and give appropriate response. This is a loop to take input from user recognize intent and give appropriate response.

Python3

print("Note: Enter 'quit' to break the loop.")
while True:
    query = input('You: ')
    if query.lower() == 'quit':
        break
    bot_response, typ = response(query)
    print('Geek: {} -- TYPE: {}'.format(bot_response, typ))
    print()

Output:

Note: Enter 'quit' to break the loop.
You:  Hi, Who are you?
Geek: Hello, how are you? I am great thanks! Please tell me your GeniSys user -- TYPE: CourtesyGreeting

You:  Can you prove you have a conscious?
Geek: That is an difficult question, can you prove that you are? -- TYPE: SelfAware

You:  Can you prove you are self-aware?
Geek: They call you <HUMAN>, what can I do for you? -- TYPE: CurrentHumanQuery

You:  My name is Pawan Gunjan.
Geek: <HUMAN>, what can I do for you? -- TYPE: CurrentHumanQuery

You:  Tell me a Jokes
Geek: A woman goes into a US sporting goods store to buy a rifle. 'It's for my husband', she tells the clerk. 'Did he tell you what gauge to get?' asks the clerk. Are you kidding?' she says. 'He doesn't even know that I'm going to shoot him!' -- TYPE: Jokes

You:  Tell me aJokes
Geek: Let me see -- TYPE: WhoAmI

You:  Tell me a Jokes
Geek: A team of scientists were nominated for the Nobel Prize. They had used dental equipment to discover and measure the smallest particles yet known to man. They became known as 'The Graders of the Flossed Quark...' -- TYPE: Jokes

You:  quit