Open In App

Sign Language Recognition System using TensorFlow in Python

Improve
Improve
Like Article
Like
Save
Share
Report

The first step of any machine learning problem is finding the appropriate dataset. For Sign language recognition let’s use the Sign Language MNIST dataset. It has images of signs corresponding to each alphabet in the English language. Since the sign language of J and Z requires motion, those two classes are not available in the dataset. 

Importing Libraries

Python libraries make it very easy for us to handle the data and perform typical and complex tasks with a single line of code.

  • Pandas – This library helps to load the data frame in a 2D array format and has multiple functions to perform analysis tasks in one go.
  • Numpy – Numpy arrays are very fast and can perform large computations in a very short time.
  • Matplotlib – This library is used to draw visualizations.
  • Tensorflow – This is an open-source library that is used for Machine Learning and Artificial intelligence and provides a range of functions to achieve complex functionalities with single lines of code.

Python3




import string
import pandas as pd
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from tensorflow.keras.preprocessing.image import ImageDataGenerator


Importing Dataset

The dataset is available as two CSV files, sign_mnist_train.csv and sign_mnist_test.csv. Each row in the CSV file is a training sample with the 0th index having the labels from 0-25 and the rest of the row containing the 784-pixel values of a 28 x 28 image. Each pixel value will be in the range [0, 255].

Python3




df = pd.read_csv('/content/sign_mnist_train.csv')
df.head()


First five rows of the dataset

First five rows of the dataset

Data Loading and Preprocessing

The dataset has been provided in two files one is for training and the other one is for testing. We will load this data and then one hot encode the labels considering the fact we are not building the classifier for ‘J’ and ‘Z’ alphabet.

Python3




def load_data(path):
    df = pd.read_csv(path)
    y = np.array([label if label < 9
                  else label-1 for label in df['label']])
    df = df.drop('label', axis=1)
    x = np.array([df.iloc[i].to_numpy().reshape((28, 28))
                  for i in range(len(df))]).astype(float)
    x = np.expand_dims(x, axis=3)
    y = pd.get_dummies(y).values
  
    return x, y
  
X_train, Y_train = load_data('/content/sign_mnist_train.csv')
X_test, Y_test = load_data('/content/sign_mnist_test.csv')


Now let’s check the shape of the training and the testing data.

Python3




print(X_train.shape, Y_train.shape)
print(X_test.shape, Y_test.shape)


Output:

(27455, 28, 28, 1) (27455, 24)
(7172, 28, 28, 1) (7172, 24)

Data Visualization

In this section, we will try to visualize images for signs of some of the alphabets which have been provided to us to build the classifier for each class.

Python3




class_names = list(string.ascii_lowercase[:26].replace(
    'j', '').replace('z', ''))
  
plt.figure(figsize=(10, 10))
for i in range(10):
    plt.subplot(5, 5, i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(X_train[i].squeeze(), cmap=plt.cm.binary)
    plt.xlabel(class_names[np.argmax(Y_train, axis=1)[i]])
plt.tight_layout()
plt.show()


Output:

Visualizing Some images from the training data

Visualizing Some images from the training data

Model Development

From this step onward we will use the TensorFlow library to build our CNN model. Keras framework of the tensor flow library contains all the functionalities that one may need to define the architecture of a Convolutional Neural Network and train it on the data.

Model Architecture

We will implement a Sequential model which will contain the following parts:

  • Three Convolutional Layers followed by MaxPooling Layers.
  • The Flatten layer to flatten the output of the convolutional layer.
  • Then we will have two fully connected layers followed by the output of the flattened layer.
  • We have included some BatchNormalization layers to enable stable and fast training and a Dropout layer before the final layer to avoid any possibility of overfitting.
  • The final layer is the output layer which outputs soft probabilities for the 24 classes. 

Python3




model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(filters=32,
                           kernel_size=(3, 3),
                           activation='relu',
                           input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(2, 2),
  
    tf.keras.layers.Conv2D(filters=64,
                           kernel_size=(3, 3),
                           activation='relu'),
    tf.keras.layers.MaxPooling2D(2, 2),
  
    tf.keras.layers.Flatten(),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(24, activation='softmax')
])


Let’s print the summary of the model’s architecture:

Python3




model.summary()


Output:

Summary of the model architecture

Summary of the model architecture

While compiling a model we provide these three essential parameters:

  • optimizer – This is the method that helps to optimize the cost function by using gradient descent.
  • loss – The loss function by which we monitor whether the model is improving with training or not.
  • metrics – This helps to evaluate the model by predicting the training and the validation data.

Python3




model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)


Now we will train our model:

Python3




history = model.fit(train_generator,
                    validation_data=val_generator,
                    epochs=5,
                    verbose=1)


Output:

Progress of the model training

Progress of the model training

Model Evaluation

Let’s visualize the training and validation accuracy with each epoch.

Python3




history_df = pd.DataFrame(history.history)
history_df.loc[:,['loss','val_loss']].plot()
history_df.loc[:,['accuracy','val_accuracy']].plot()
plt.show()


Output:

The plot of loss and accuracy epoch by epoch for training as well as validation data.

The plot of loss and accuracy epoch by epoch for training as well as validation data.

Python3




model.evaluate(val_generator)


Output:

Evaluating model's performance on validation data

Evaluating the model’s performance on validation data

Conclusion:

By using just a simple CNN model we are able to achieve an accuracy of 82% which is really great. This shows that this technology is certainly going to help us build some amazing applications which can be proved a really great tool for people with some special needs.



Last Updated : 30 Dec, 2022
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads