Let us discuss gesture-controlled game using TensorFlow, I hope this article will be much interesting as it provides a holistic gaming experience at the end. We begin by describing the modules necessary to develop this game.
Necessary Libraries
We start our coding by importing the necessary libraries, including cv2 for capturing video from the webcam, then mediapipe for hand tracking, also numpy for numerical operations, then Tensorflow for loading the pre-trained model and pygame for creating the game window and handling game-related functionalities.
pip install cv2 mediapipe numpy tensorflow pygame
Below is aby-step step-wise approach for developing gesture controlled game using TensorFlow
STEP 1: Import all the necessary libraries,
For this gesture-controlled game five libraries are required.
import cv2
import mediapipe as mp
import numpy as np
import tensorflow as tf
import pygame
|
Output:
pygame 2.5.0 (SDL 2.28.0, Python 3.10.6)
Hello from the pygame community. https://www.pygame.org/contribute.html
STEP 2: Loading the pre-trained model:
The pre-trained TensorFlow model is loaded using tf.keras.models.load_model(). This model is trained to recognize hand gestures based on hand landmarks. Add “gesture_model.h5” inside the same folder. You can download the pre-trained gesture model using this GitHub link: gesture_model.h5
# Load the pre-trained TensorFlow model model = tf.keras.models.load_model( 'get_gesture_model/gesture_model.h5' )
model.summary() |
Output:
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 100, 100, 8) 136
max_pooling2d_1 (MaxPooling (None, 24, 24, 8) 0
2D)
conv2d_2 (Conv2D) (None, 24, 24, 16) 528
max_pooling2d_2 (MaxPooling (None, 12, 12, 16) 0
2D)
flatten_1 (Flatten) (None, 2304) 0
dense_1 (Dense) (None, 512) 1180160
dropout_1 (Dropout) (None, 512) 0
dense_2 (Dense) (None, 128) 65664
dropout_2 (Dropout) (None, 128) 0
dense_3 (Dense) (None, 10) 1290
=================================================================
Total params: 1,247,778
Trainable params: 1,247,778
Non-trainable params: 0
_________________________________________________________________
STEP 3: Initialize MediaPipe Hands model:
The below code initializes the MediaPipe’s Hands Model by creating an instance of mp_hands.Hands. This model is used for detecting hand landmarks in each frame captured from the webcam. From the gesture_labels dictionary, it takes 0 as ‘Left’ and 1 as ‘Right’
# Initialize MediaPipe's Hands model mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode = False , max_num_hands = 1 , min_detection_confidence = 0.5 )
# Gesture labels gesture_labels = {
0 : 'Left' ,
1 : 'Right' ,
} |
STEP 4: Initialize Pygame, Setting up Player and Game Loop:
Next Pygame is initialized by calling pygame.init(). The game window dimensions are set, and the window caption is set to “Gesture-Based Game”. The player’s properties, such as width, height, initial position, and speed, are defined clearly. The player is represented as a white rectangle. The main game loop is set up using a while loop. This loop continues until the game is quit which means the player hits the obstacle.
# Initialize Pygame pygame.init() window_width = 800
window_height = 600
window = pygame.display.set_mode((window_width, window_height))
pygame.display.set_caption( "Gesture-Based Game" )
# Set up the player player_width = 50
player_height = 50
player_x = window_width / / 2 - player_width / / 2
player_y = window_height - player_height - 10
player_speed = 5
# Set up the obstacle obstacle_width = 50
obstacle_height = 50
obstacle_x = np.random.randint( 0 , window_width - obstacle_width)
obstacle_y = 0
obstacle_speed = 3
# Game over flag and text game_over = False
game_over_text = pygame.font.Font( 'freesansbold.ttf' , 64 ).render( 'Game Over' , True , ( 255 , 255 , 255 ))
# Restart button restart_button_text = pygame.font.Font( 'freesansbold.ttf' , 32 ).render( 'Restart' , True , ( 255 , 255 , 255 ))
restart_button_rect = restart_button_text.get_rect()
restart_button_rect.center = (window_width / / 2 , window_height / / 2 + 50 )
# Game loop running = True
clock = pygame.time.Clock()
|
STEP 5: Control with live video with finger
- Capturing video from webcam: Video frames are captured from the webcam using cap.read(). The frame is flipped horizontally to create a mirror effect. The captured live gestures are recognized to perform the left and right movements of player.
- Detecting hand landmarks and recognizing gestures: Then this code processes each frame using the MediaPipe Hands model to detect hand landmarks. It specifically tracks the index finger’s position to recognize gestures. Based on the position of the index finger, the code determines whether the gesture is “Left”, “Right”, or “Neutral”.
- Updating the player’s position: In this code,, the player’s position is updated based on the recognized gesture. If the gesture is “Left,” the player moves left by subtracting the player’s speed from its x-coordinate. If the gesture is “Right,” the player moves right by adding the player speed to its x-coordinate. If the gesture is “Neutral,” the player remains stationary. Using this we can handle the player with ease.
- Creating player and obstacle: Both the player and obstacle are generated inside pygame using pygame.draw.rect(). If both the player and obstacle hits in the game, then it displays “Game Over” along with a Restart button to continue playing the game again.In order to quit the game, press the close window or press X button. It close all windows and quits the pygame instance.
# Capturing video from webcam: cap = cv2.VideoCapture( 0 )
while running:
for event in pygame.event.get():
if event. type = = pygame.QUIT:
running = False
if event. type = = pygame.MOUSEBUTTONDOWN and game_over:
mouse_pos = pygame.mouse.get_pos()
if restart_button_rect.collidepoint(mouse_pos):
# Restart the game
game_over = False
player_x = window_width / / 2 - player_width / / 2
obstacle_x = np.random.randint( 0 , window_width - obstacle_width)
obstacle_y = 0
success, frame = cap.read()
if not success:
break
# Flip the frame horizontally for a mirror effect
frame = cv2.flip(frame, 1 )
# Detecting hand landmarks and recognizing gestures results = hands.process(frame)
if results.multi_hand_landmarks:
for hand_landmarks in results.multi_hand_landmarks:
# Get the index finger landmarks
index_finger_landmarks = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
index_finger_x = int (index_finger_landmarks.x * frame.shape[ 1 ])
index_finger_y = int (index_finger_landmarks.y * frame.shape[ 0 ])
# Recognize gesture based on index finger position
if index_finger_x < frame.shape[ 1 ] / / 3 :
gesture = 'Left'
player_x - = player_speed
elif index_finger_x > frame.shape[ 1 ] * 2 / / 3 :
gesture = 'Right'
player_x + = player_speed
else :
gesture = 'Neutral'
# Draw a circle at the index finger position
cv2.circle(frame, (index_finger_x, index_finger_y), 10 , ( 0 , 255 , 0 ), - 1 )
# Draw the recognized gesture on the frame
cv2.putText(frame, gesture, ( 10 , 30 ), cv2.FONT_HERSHEY_SIMPLEX, 1 , ( 0 , 0 , 255 ), 2 )
cv2.imshow( 'Gesture Recognition' , frame)
if cv2.waitKey( 1 ) & 0xFF = = ord ( 'q' ):
break
# Updating the player's position if not game_over:
# Update the player's position
if player_x < 0 :
player_x = 0
elif player_x > window_width - player_width:
player_x = window_width - player_width
# Update the obstacle's position
obstacle_y + = obstacle_speed
# Check for collision between the player and the obstacle
if player_x < obstacle_x + obstacle_width and \
player_x + player_width > obstacle_x and \
player_y < obstacle_y + obstacle_height and \
player_y + player_height > obstacle_y:
# Game over
game_over = True
# Reset the obstacle if it goes off the screen
if obstacle_y > window_height:
obstacle_x = np.random.randint( 0 , window_width - obstacle_width)
obstacle_y = 0
# Clear the window
window.fill(( 0 , 0 , 0 ))
# Creating player and obstacle: pygame.draw.rect(window, ( 255 , 255 , 255 ), (player_x, player_y, player_width, player_height))
# Draw the obstacle
pygame.draw.rect(window, ( 255 , 0 , 0 ), (obstacle_x, obstacle_y, obstacle_width, obstacle_height))
# Display "Game Over" text if game over
if game_over:
window.blit(game_over_text,
(window_width / / 2 - game_over_text.get_width() / / 2 ,
window_height / / 2 - game_over_text.get_height() / / 2 ))
pygame.draw.rect(window, ( 0 , 0 , 255 ), restart_button_rect)
window.blit(restart_button_text, restart_button_rect)
# Update the display
pygame.display.update()
# Limit the frame rate
clock.tick( 60 )
cap.release() cv2.destroyAllWindows() pygame.quit() |
Full Code Implementation of Gesture Controlled Game :
Here the entire code, combining all the above steps to start playing our gesture controlled game using TensorFlow is ready. The below code combines video processing, gesture recognition, and game mechanics in order to create an interactive gesture-based gaming experience. Further, improve this code with additional features, and more gestures to make this very interesting.Let’s start the game now.
How to play this gesture-based game
- Launch the game: Run the Python code to start the game. The game window will open, displaying a white rectangular player at the bottom of the screen.
- Position your hand: Position your hand in front of the webcam, ensuring it is within the camera’s field of view. Make sure your hand is well-lit and distinguishable from the background.
- Recognize gestures: The game uses hand gestures to control the player’s movement. Extend your index finger to control the player’s movement.
- Left gesture: Move your extended index finger to the left side of the screen to make the player move left.
- Right gesture: Move your extended index finger to the right side of the screen to make the player move right.
- Neutral gesture: If your index finger is in the middle area of the screen, the player will stay in its current position.
- Dodge the falling obstacle: An obstacle in the form of a red rectangular box will fall from the top of the screen. You aim to dodge the obstacle by moving the player left or right using hand gestures.
- Game over: If the player collides with the falling obstacle, the game ends. The screen will display a “Game Over” message, along with a restart button.
- Restart the game: To restart the game, simply click the restart button displayed on the screen when the game is over. The player and obstacle positions will be reset, and you can continue playing.
- Quit the game: To quit the game, close the game window or press the “X” button.
import cv2
import mediapipe as mp
import numpy as np
import tensorflow as tf
import pygame
# Load the pre-trained TensorFlow model model = tf.keras.models.load_model( 'gesture_model.h5' )
# Initialize MediaPipe's Hands model mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode = False , max_num_hands = 1 , min_detection_confidence = 0.5 )
# Gesture labels gesture_labels = {
0 : 'Left' ,
1 : 'Right' ,
} # Initialize Pygame pygame.init() window_width = 800
window_height = 600
window = pygame.display.set_mode((window_width, window_height))
pygame.display.set_caption( "Gesture-Based Game" )
# Set up the player player_width = 50
player_height = 50
player_x = window_width / / 2 - player_width / / 2
player_y = window_height - player_height - 10
player_speed = 5
# Set up the obstacle obstacle_width = 50
obstacle_height = 50
obstacle_x = np.random.randint( 0 , window_width - obstacle_width)
obstacle_y = 0
obstacle_speed = 3
# Game over flag and text game_over = False
game_over_text = pygame.font.Font( 'freesansbold.ttf' , 64 ).render( 'Game Over' , True , ( 255 , 255 , 255 ))
# Restart button restart_button_text = pygame.font.Font( 'freesansbold.ttf' , 32 ).render( 'Restart' , True , ( 255 , 255 , 255 ))
restart_button_rect = restart_button_text.get_rect()
restart_button_rect.center = (window_width / / 2 , window_height / / 2 + 50 )
# Game loop running = True
clock = pygame.time.Clock()
# Main loop cap = cv2.VideoCapture( 0 )
while running:
for event in pygame.event.get():
if event. type = = pygame.QUIT:
running = False
if event. type = = pygame.MOUSEBUTTONDOWN and game_over:
mouse_pos = pygame.mouse.get_pos()
if restart_button_rect.collidepoint(mouse_pos):
# Restart the game
game_over = False
player_x = window_width / / 2 - player_width / / 2
obstacle_x = np.random.randint( 0 , window_width - obstacle_width)
obstacle_y = 0
success, frame = cap.read()
if not success:
break
# Flip the frame horizontally for a mirror effect
frame = cv2.flip(frame, 1 )
# Detect hand landmarks
results = hands.process(frame)
if results.multi_hand_landmarks:
for hand_landmarks in results.multi_hand_landmarks:
# Get the index finger landmarks
index_finger_landmarks = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
index_finger_x = int (index_finger_landmarks.x * frame.shape[ 1 ])
index_finger_y = int (index_finger_landmarks.y * frame.shape[ 0 ])
# Recognize gesture based on index finger position
if index_finger_x < frame.shape[ 1 ] / / 3 :
gesture = 'Left'
player_x - = player_speed
elif index_finger_x > frame.shape[ 1 ] * 2 / / 3 :
gesture = 'Right'
player_x + = player_speed
else :
gesture = 'Neutral'
# Draw a circle at the index finger position
cv2.circle(frame, (index_finger_x, index_finger_y), 10 , ( 0 , 255 , 0 ), - 1 )
# Draw the recognized gesture on the frame
cv2.putText(frame, gesture, ( 10 , 30 ), cv2.FONT_HERSHEY_SIMPLEX, 1 , ( 0 , 0 , 255 ), 2 )
cv2.imshow( 'Gesture Recognition' , frame)
if cv2.waitKey( 1 ) & 0xFF = = ord ( 'q' ):
break
if not game_over:
# Update the player's position
if player_x < 0 :
player_x = 0
elif player_x > window_width - player_width:
player_x = window_width - player_width
# Update the obstacle's position
obstacle_y + = obstacle_speed
# Check for collision between the player and the obstacle
if player_x < obstacle_x + obstacle_width and \
player_x + player_width > obstacle_x and \
player_y < obstacle_y + obstacle_height and \
player_y + player_height > obstacle_y:
# Game over
game_over = True
# Reset the obstacle if it goes off the screen
if obstacle_y > window_height:
obstacle_x = np.random.randint( 0 , window_width - obstacle_width)
obstacle_y = 0
# Clear the window
window.fill(( 0 , 0 , 0 ))
# Draw the player
pygame.draw.rect(window, ( 255 , 255 , 255 ), (player_x, player_y, player_width, player_height))
# Draw the obstacle
pygame.draw.rect(window, ( 255 , 0 , 0 ), (obstacle_x, obstacle_y, obstacle_width, obstacle_height))
# Display "Game Over" text if game over
if game_over:
window.blit(game_over_text, (window_width / / 2 - game_over_text.get_width() / / 2 , window_height / / 2 - game_over_text.get_height() / / 2 ))
pygame.draw.rect(window, ( 0 , 0 , 255 ), restart_button_rect)
window.blit(restart_button_text, restart_button_rect)
# Update the display
pygame.display.update()
# Limit the frame rate
clock.tick( 60 )
cap.release() cv2.destroyAllWindows() pygame.quit() |
Output:
pygame 2.5.0 (SDL 2.28.0, Python 3.10.6)
Hello from the pygame community. https://www.pygame.org/contribute.html
Launch Gesture Recognition and Gesture Based Game
Pygame starts the game with the player and obstacle, and the gesture recognition shows Neutral to perform the player action.
Screenshot 3: Displays Game Over and Option to Restart the game.
The player can restart the game with restart button, to continue playing endlessly.
Feel free to support me with your comments below. Happy Playing!