Gesture Controlled Game in Machine Learning

Let us discuss gesture-controlled game using TensorFlow, I hope this article will be much interesting as it provides a holistic gaming experience at the end. We begin by describing the modules necessary to develop this game.

Necessary Libraries

We start our coding by importing the necessary libraries, including cv2 for capturing video from the webcam, then mediapipe for hand tracking, also numpy for numerical operations, then Tensorflow for loading the pre-trained model and pygame for creating the game window and handling game-related functionalities.

• OpenCV
• Mediapipe
• Numpy
• TensorFlow
• Pygame

pip install cv2 mediapipe numpy tensorflow pygame

Below is aby-step step-wise approach for developing gesture controlled game using TensorFlow

STEP 1: Import all the necessary libraries,

For this gesture-controlled game five libraries are required.

Python3

import cv2
import mediapipe as mp
import numpy as np
import tensorflow as tf
import pygame

Output:

pygame 2.5.0 (SDL 2.28.0, Python 3.10.6)
Hello from the pygame community. https://www.pygame.org/contribute.html

STEP 2: Loading the pre-trained model:

The pre-trained TensorFlow model is loaded using tf.keras.models.load_model(). This model is trained to recognize hand gestures based on hand landmarks. Add “gesture_model.h5” inside the same folder. You can download the pre-trained gesture model using this GitHub link: gesture_model.h5

Python3

# Load the pre-trained TensorFlow model
model = tf.keras.models.load_model('get_gesture_model/gesture_model.h5')
model.summary()

Output:

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_1 (Conv2D)           (None, 100, 100, 8)       136       
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 24, 24, 8)        0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 24, 24, 16)        528       
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 12, 12, 16)       0         
 2D)                                                             
                                                                 
 flatten_1 (Flatten)         (None, 2304)              0         
                                                                 
 dense_1 (Dense)             (None, 512)               1180160   
                                                                 
 dropout_1 (Dropout)         (None, 512)               0         
                                                                 
 dense_2 (Dense)             (None, 128)               65664     
                                                                 
 dropout_2 (Dropout)         (None, 128)               0         
                                                                 
 dense_3 (Dense)             (None, 10)                1290      
                                                                 
=================================================================
Total params: 1,247,778
Trainable params: 1,247,778
Non-trainable params: 0
_________________________________________________________________

STEP 3: Initialize MediaPipe Hands model:

The below code initializes the MediaPipe’s Hands Model by creating an instance of mp_hands.Hands. This model is used for detecting hand landmarks in each frame captured from the webcam. From the gesture_labels dictionary, it takes 0 as ‘Left’ and 1 as ‘Right’

Python3

# Initialize MediaPipe's Hands model
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode=False, max_num_hands=1, min_detection_confidence=0.5)

# Gesture labels
gesture_labels = {
    0: 'Left',
    1: 'Right',
}

STEP 4: Initialize Pygame, Setting up Player and Game Loop:

Next Pygame is initialized by calling pygame.init(). The game window dimensions are set, and the window caption is set to “Gesture-Based Game”. The player’s properties, such as width, height, initial position, and speed, are defined clearly. The player is represented as a white rectangle. The main game loop is set up using a while loop. This loop continues until the game is quit which means the player hits the obstacle.

Python3

# Initialize Pygame
pygame.init()
window_width = 800
window_height = 600
window = pygame.display.set_mode((window_width, window_height))
pygame.display.set_caption("Gesture-Based Game")

# Set up the player
player_width = 50
player_height = 50
player_x = window_width // 2 - player_width // 2
player_y = window_height - player_height - 10
player_speed = 5

# Set up the obstacle
obstacle_width = 50
obstacle_height = 50
obstacle_x = np.random.randint(0, window_width - obstacle_width)
obstacle_y = 0
obstacle_speed = 3

# Game over flag and text
game_over = False
game_over_text = pygame.font.Font('freesansbold.ttf', 64).render('Game Over', True, (255, 255, 255))

# Restart button
restart_button_text = pygame.font.Font('freesansbold.ttf', 32).render('Restart', True, (255, 255, 255))
restart_button_rect = restart_button_text.get_rect()
restart_button_rect.center = (window_width // 2, window_height // 2 + 50)

# Game loop
running = True
clock = pygame.time.Clock()

STEP 5: Control with live video with finger

1. Capturing video from webcam: Video frames are captured from the webcam using cap.read(). The frame is flipped horizontally to create a mirror effect. The captured live gestures are recognized to perform the left and right movements of player.
2. Detecting hand landmarks and recognizing gestures: Then this code processes each frame using the MediaPipe Hands model to detect hand landmarks. It specifically tracks the index finger’s position to recognize gestures. Based on the position of the index finger, the code determines whether the gesture is “Left”, “Right”, or “Neutral”.
3. Updating the player’s position: In this code,, the player’s position is updated based on the recognized gesture. If the gesture is “Left,” the player moves left by subtracting the player’s speed from its x-coordinate. If the gesture is “Right,” the player moves right by adding the player speed to its x-coordinate. If the gesture is “Neutral,” the player remains stationary. Using this we can handle the player with ease.
4. Creating player and obstacle: Both the player and obstacle are generated inside pygame using pygame.draw.rect(). If both the player and obstacle hits in the game, then it displays “Game Over” along with a Restart button to continue playing the game again.In order to quit the game, press the close window or press X button. It close all windows and quits the pygame instance.

Python3

# Capturing video from webcam:
cap = cv2.VideoCapture(0)
while running:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False
        if event.type == pygame.MOUSEBUTTONDOWN and game_over:
            mouse_pos = pygame.mouse.get_pos()
            if restart_button_rect.collidepoint(mouse_pos):
                # Restart the game
                game_over = False
                player_x = window_width // 2 - player_width // 2
                obstacle_x = np.random.randint(0, window_width - obstacle_width)
                obstacle_y = 0

    success, frame = cap.read()
    if not success:
        break

    # Flip the frame horizontally for a mirror effect
    frame = cv2.flip(frame, 1)

# Detecting hand landmarks and recognizing gestures
    results = hands.process(frame)
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Get the index finger landmarks
            index_finger_landmarks = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
            index_finger_x = int(index_finger_landmarks.x * frame.shape[1])
            index_finger_y = int(index_finger_landmarks.y * frame.shape[0])

            # Recognize gesture based on index finger position
            if index_finger_x < frame.shape[1] // 3:
                gesture = 'Left'
                player_x -= player_speed
            elif index_finger_x > frame.shape[1] * 2 // 3:
                gesture = 'Right'
                player_x += player_speed
            else:
                gesture = 'Neutral'

            # Draw a circle at the index finger position
            cv2.circle(frame, (index_finger_x, index_finger_y), 10, (0, 255, 0), -1)

            # Draw the recognized gesture on the frame
            cv2.putText(frame, gesture, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)

    cv2.imshow('Gesture Recognition', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
# Updating the player's position
    if not game_over:
        # Update the player's position
        if player_x < 0:
            player_x = 0
        elif player_x > window_width - player_width:
            player_x = window_width - player_width

        # Update the obstacle's position
        obstacle_y += obstacle_speed

        # Check for collision between the player and the obstacle
        if player_x < obstacle_x + obstacle_width and \
                player_x + player_width > obstacle_x and \
                player_y < obstacle_y + obstacle_height and \
                player_y + player_height > obstacle_y:
            # Game over
            game_over = True

        # Reset the obstacle if it goes off the screen
        if obstacle_y > window_height:
            obstacle_x = np.random.randint(0, window_width - obstacle_width)
            obstacle_y = 0

        # Clear the window
        window.fill((0, 0, 0))

# Creating player and obstacle:
        pygame.draw.rect(window, (255, 255, 255), (player_x, player_y, player_width, player_height))

        # Draw the obstacle
        pygame.draw.rect(window, (255, 0, 0), (obstacle_x, obstacle_y, obstacle_width, obstacle_height))

    # Display "Game Over" text if game over
    if game_over:
        window.blit(game_over_text,
                    (window_width // 2 - game_over_text.get_width() // 2,
                     window_height // 2 - game_over_text.get_height() // 2))
        pygame.draw.rect(window, (0, 0, 255), restart_button_rect)
        window.blit(restart_button_text, restart_button_rect)

    # Update the display
    pygame.display.update()

    # Limit the frame rate
    clock.tick(60)

cap.release()
cv2.destroyAllWindows()
pygame.quit()

Full Code Implementation of Gesture Controlled Game :

Here the entire code, combining all the above steps to start playing our gesture controlled game using TensorFlow is ready. The below code combines video processing, gesture recognition, and game mechanics in order to create an interactive gesture-based gaming experience. Further, improve this code with additional features, and more gestures to make this very interesting.Let’s start the game now.

How to play this gesture-based game

1. Launch the game: Run the Python code to start the game. The game window will open, displaying a white rectangular player at the bottom of the screen.
2. Position your hand: Position your hand in front of the webcam, ensuring it is within the camera’s field of view. Make sure your hand is well-lit and distinguishable from the background.
3. Recognize gestures: The game uses hand gestures to control the player’s movement. Extend your index finger to control the player’s movement.
4. Left gesture: Move your extended index finger to the left side of the screen to make the player move left.
5. Right gesture: Move your extended index finger to the right side of the screen to make the player move right.
6. Neutral gesture: If your index finger is in the middle area of the screen, the player will stay in its current position.
7. Dodge the falling obstacle: An obstacle in the form of a red rectangular box will fall from the top of the screen. You aim to dodge the obstacle by moving the player left or right using hand gestures.
8. Game over: If the player collides with the falling obstacle, the game ends. The screen will display a “Game Over” message, along with a restart button.
9. Restart the game: To restart the game, simply click the restart button displayed on the screen when the game is over. The player and obstacle positions will be reset, and you can continue playing.
10. Quit the game: To quit the game, close the game window or press the “X” button.

Python3

import cv2
import mediapipe as mp
import numpy as np
import tensorflow as tf
import pygame

# Load the pre-trained TensorFlow model
model = tf.keras.models.load_model('gesture_model.h5')

# Initialize MediaPipe's Hands model
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode=False, max_num_hands=1, min_detection_confidence=0.5)

# Gesture labels
gesture_labels = {
    0: 'Left',
    1: 'Right',
}

# Initialize Pygame
pygame.init()
window_width = 800
window_height = 600
window = pygame.display.set_mode((window_width, window_height))
pygame.display.set_caption("Gesture-Based Game")

# Set up the player
player_width = 50
player_height = 50
player_x = window_width // 2 - player_width // 2
player_y = window_height - player_height - 10
player_speed = 5

# Set up the obstacle
obstacle_width = 50
obstacle_height = 50
obstacle_x = np.random.randint(0, window_width - obstacle_width)
obstacle_y = 0
obstacle_speed = 3

# Game over flag and text
game_over = False
game_over_text = pygame.font.Font('freesansbold.ttf', 64).render('Game Over', True, (255, 255, 255))

# Restart button
restart_button_text = pygame.font.Font('freesansbold.ttf', 32).render('Restart', True, (255, 255, 255))
restart_button_rect = restart_button_text.get_rect()
restart_button_rect.center = (window_width // 2, window_height // 2 + 50)

# Game loop
running = True
clock = pygame.time.Clock()

# Main loop
cap = cv2.VideoCapture(0)
while running:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False
        if event.type == pygame.MOUSEBUTTONDOWN and game_over:
            mouse_pos = pygame.mouse.get_pos()
            if restart_button_rect.collidepoint(mouse_pos):
                # Restart the game
                game_over = False
                player_x = window_width // 2 - player_width // 2
                obstacle_x = np.random.randint(0, window_width - obstacle_width)
                obstacle_y = 0

    success, frame = cap.read()
    if not success:
        break

    # Flip the frame horizontally for a mirror effect
    frame = cv2.flip(frame, 1)

    # Detect hand landmarks
    results = hands.process(frame)
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Get the index finger landmarks
            index_finger_landmarks = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
            index_finger_x = int(index_finger_landmarks.x * frame.shape[1])
            index_finger_y = int(index_finger_landmarks.y * frame.shape[0])

            # Recognize gesture based on index finger position
            if index_finger_x < frame.shape[1] // 3:
                gesture = 'Left'
                player_x -= player_speed
            elif index_finger_x > frame.shape[1] * 2 // 3:
                gesture = 'Right'
                player_x += player_speed
            else:
                gesture = 'Neutral'

            # Draw a circle at the index finger position
            cv2.circle(frame, (index_finger_x, index_finger_y), 10, (0, 255, 0), -1)

            # Draw the recognized gesture on the frame
            cv2.putText(frame, gesture, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)

    cv2.imshow('Gesture Recognition', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

    if not game_over:
        # Update the player's position
        if player_x < 0:
            player_x = 0
        elif player_x > window_width - player_width:
            player_x = window_width - player_width

        # Update the obstacle's position
        obstacle_y += obstacle_speed

        # Check for collision between the player and the obstacle
        if player_x < obstacle_x + obstacle_width and \
                player_x + player_width > obstacle_x and \
                player_y < obstacle_y + obstacle_height and \
                player_y + player_height > obstacle_y:
            # Game over
            game_over = True

        # Reset the obstacle if it goes off the screen
        if obstacle_y > window_height:
            obstacle_x = np.random.randint(0, window_width - obstacle_width)
            obstacle_y = 0

        # Clear the window
        window.fill((0, 0, 0))

        # Draw the player
        pygame.draw.rect(window, (255, 255, 255), (player_x, player_y, player_width, player_height))

        # Draw the obstacle
        pygame.draw.rect(window, (255, 0, 0), (obstacle_x, obstacle_y, obstacle_width, obstacle_height))

    # Display "Game Over" text if game over
    if game_over:
        window.blit(game_over_text, (window_width // 2 - game_over_text.get_width() // 2, window_height // 2 - game_over_text.get_height() // 2))
        pygame.draw.rect(window, (0, 0, 255), restart_button_rect)
        window.blit(restart_button_text, restart_button_rect)

    # Update the display
    pygame.display.update()

    # Limit the frame rate
    clock.tick(60)

cap.release()
cv2.destroyAllWindows()
pygame.quit()

Output:

pygame 2.5.0 (SDL 2.28.0, Python 3.10.6)
Hello from the pygame community. https://www.pygame.org/contribute.html

Launch Gesture Recognition and Gesture Based Game

Pygame starts the game with the player and obstacle, and the gesture recognition shows Neutral to perform the player action.

Gesture Recognition – Neutral

Screenshot 3: Displays Game Over and Option to Restart the game.

The player can restart the game with restart button, to continue playing endlessly.

Game Over and Restart

Feel free to support me with your comments below. Happy Playing!