Project Idea – Object Detection and Tracking

Project title: Object Detection and Tracking

Introduction: A lot of people go to supermarkets and retail stores and shops to idle around and window-shop instead of purchasing any products. The thought of analyzing this kind of behavior was intriguing.  

• How does this kind of behavior affect product sales?
• What time periods these people were coming in?
• What could help the owners count the number of customers by cross-referencing the billing data, but how do you count the people who haven’t shopped?

Object detection and tracking is one of the areas of computer vision that is maturing very rapidly. It allows us to identify and locate objects in an image or video. With this kind of identification and localization, object detection and tracking can be used to count objects in a particular scene and determine and track their precise locations, all while accurately labeling them.

In this project, we have made use of two of the most popular Python libraries for object detection, OpenCV and ImageAI.  

Every supermarket nowadays has at least one CCTV camera installed. And the data is stored in a centralized repository with a timestamp. Our end goal was to identify the people coming in and going out of the supermarket or retail store, and categorize them under the labels “customer” or “not a customer”. By achieving this goal we could calculate the actual cost per customer.    

Methodologies

Our workflow was divided into two main segments,

1. Optimization of the videos.
2. Human Object Detection and tracking on videos.

Optimization of the videos

Optimizing the video to make it as light as possible for the algorithm to work smoothly and achieve speed.  A video is nothing but a sequence of images, hence achieving the objective of identifying the object at the lowest-most level of an image will result in achieving the same on a video by iterating the process on all the images that the video is made up of.

Optimization includes :

• Turning the image/video into grayscale
• Reducing the image/video to the pure black white form
• Masking the image/video
• Reading video frames

Human Object Detection & Tracking on videos

For object detection and tracking, we used OpenCV and ImageAI. The object detection and tracking work for recorded videos as well as a live feed directly from different types of cameras.  

Object detection and tracking include:

• Using the camera for live-feed video
• Using existing video footage
• An in-out tracker using opencv and object detection and counter method
• Finally a web app as a GUI for the analysis of the detection and tracking results of the supermarket and retail stores that is done using streamlit.

Object detection and tracking method

We are accepting a set of bounding boxes of a person and compute their respective centroids and then compute the Euclidean distance between any new centroids and existing centroids to track the movement.

When centroids intercept the gate line from the top of the frame the IN counter is incremented and when the centroids intercept from down the frame OUT counter is incremented. And hence we can count the number of people in the defined zone.

Tools and Technologies Used in the Project

• Numpy: The fundamental package for scientific computing with Python.
• OpenCV: “ Open Source Computer Vision Library ” is an open-source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications.
• ImageAI: It is a Python library built to empower developers, researchers, and students to build applications and systems with self-contained Deep Learning and Computer Vision capabilities using simple and few lines of code. ImageAI provides the three most powerful models for object detection and tracking – RetinaNet, YOLOv3, and TinyYOLOv3.
  • In our project, we have used YOLOv3 as it gives a moderate performance with accuracy and moderate detection speed and time.
• Streamlit: It is an open-source Python library that makes it easy to create and share beautiful, custom web apps for machine learning and data science.

Step-by-Step Implementation

1. Install the requirements
   1. pip
   2. Opencv-python
   3. Tensorflow 2.4.0
   4. ImageAI
   5. streamlit
   6. Other dependencies: keras==2.4.3 numpy==1.19.3 pillow==7.0.0 scipy==1.4.1 h5py==2.10.0 matplotlib==3.3.2 keras-resnet==0.2.0
2. Download YOLOV3 model
3. Run the code to stream the webapp

Command to run the code :

> streamlit run webapp.py [webapp.py – main file name]

Test Cases to Demonstrate the Output

We decided to check our program on a dummy video, to check if it could count the people coming in and going out of the shops/malls/marts. The system identified the centroids and based on the direction they crossed the gate line, the counter was updated.

Ideal Output:

IN : 7
OUT: 3

Actual Output:

IN : 7
OUT: 3

As presented in the video, we can see that our program does a good job of counting the number of people accurately. (we didn’t find a lot of videos to test our program on different videos.)

Program Flow 

Project Application in Real-Life

The project can be implemented in various fields, we thought of implementing it in local marts. The objective was simple:

We talk to the owners, and they let us use their computers to run our program on their CCTV footage. After we did that, making an excel sheet of the same and doing some basic visualizations to help the owners figure out the foot of people based on the time of the day. This will help them manipulate the prices accordingly and gain more profits while providing the customers something in return as well.

Example: They can give a discount that is limited to that very specific time period, they make a profit during the times they weren’t and the customers get the products at a cheaper rate than usual. It’s a win-win situation for both.

The application can be applied for various other fields as well, say for example security systems for shops and in public places. The system can be trained to identify threats and inform the nearest authority as an alert + record the event as proof.

Project Team –

Tanya Ranjan, Rishab Sarkar, Jesal Jadeja , Bartolomeu Carvalho