In this article, we’ll learn how to implement Region proposal object detection with OpenCV, Keras and TensorFlow.
Install all the dependencies
Use the pip command for installing all the dependencies
pip install tensorflow keras imutils
pip install opencv-contrib-python
Note: Make sure about installing the above OpenCV package otherwise you might face import errors.
Step 1: Read the image and Apply the selective search method of OpenCV
In this step, we’ll read the image and apply the selective search method from OpenCV to it. This method will return as a list of rectangles which are basically the region of interest. OpenCV provides us with two different methods for this selective search, one is the “FASTER” method and the other is the “Accurate” method, you have to decide which to use depending on your use case.
Now that we have the rectangles before we go further let’s try to visualize what regions of interest it returns.
Python3
import numpy as np
import cv2
from tensorflow.keras.applications import Xception
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications import imagenet_utils
from tensorflow.keras.preprocessing.image import img_to_array
from imutils.object_detection import non_max_suppression
img = cv2.imread( 'Assets/img2.jpg' )
search = cv2.ximgproc.segmentation.createSelectiveSearchSegmentation()
search.setBaseImage(img)
search.switchToSelectiveSearchFast()
rects = search.process()
roi = img.copy()
for (x, y, w, h) in rects:
if (w / float (W) < 0.1 or h / float (H) < 0.1 ):
continue
cv2.rectangle(roi, (x, y), (x + w, y + h),
( 0 , 200 , 0 ), 2 )
roi = cv2.resize(roi, ( 640 , 640 ))
final = cv2.hconcat([cv2.resize(img, ( 640 , 640 )), roi])
cv2.imshow( 'ROI' , final)
cv2.waitKey( 0 )
|
Output:
These are all the Region of Interest that our function receives after filtering out the ROIs which are not sufficiently large, that is to say if the ROI has less than 10% of width or height than that of the image than we won’t consider it.
Step 2: Create a List of Final Input Arrays and Bounding boxes using ROI
We’ll create two separate lists which contain the image in RGB format and another list will have the bounding box coordinates. These lists will be used for prediction and creating bounding boxes respectively. We’ll also make sure that we only make predictions on sufficiently large ROI, say which have at least 20% width or height of our image.
Python3
rois = []
boxes = []
(H, W) = img.shape[: 2 ]
rois = []
boxes = []
(H, W) = img.shape[: 2 ]
for (x, y, w, h) in rects:
if w / float (W) < 0.2 or h / float (H) < 0.2 :
continue
roi = img[y:y + h, x:x + w]
roi = cv2.cvtColor(roi, cv2.COLOR_BGR2RGB)
roi = cv2.resize(roi, ( 299 , 299 ))
roi = img_to_array(roi)
roi = preprocess_input(roi)
rois.append(roi)
x1, y1, x2, y2 = x, y, x + w, y + h
boxes.append((x1, y1, x2, y2))
|
Now that we have our region of interests which we have filtered and also preprocessed, let’s use them to create predictions using our model.
Step 3: Generate Predictions using Model
We’re using the ResNet50 model from the Keras pre-trained models, mainly because it’s not heavy on the machine and also has a high accuracy. So, first, we’ll create our model instance and then pass in our input -> List of ROIs and generate predictions.
In code it would look like this:
# ———— Model————— #
model = Xception(weights=’imagenet’)
# Convert ROIS list to arrays for predictions
input_array = np.array(rois)
print(“Input array shape is ;” ,input_array.shape)
#———- Make Predictions ——-#
preds = model.predict(input_array)
preds = imagenet_utils.decode_predictions(preds, top=1)
Now, that we have predictions let’s s show the result on the image.
Step 4: Create Objects Dictionary
In this step, we’ll create a new dictionary that basically contains the label as the key as the bounding box, and probability as the values. This we’ll easily let us access the predictions for each label and apply non_max_suppression to them. We can do this by looping through the predictions and filtering out the predictions with more than 90% confidence( you can change it to your needs). Let’s see the code:
Python3
objects = {}
for (i, pred) in enumerate (preds):
iD = pred[ 0 ][ 0 ]
label = pred[ 0 ][ 1 ]
prob = pred[ 0 ][ 2 ]
if prob > = 0.9 :
box = boxes[i]
value = objects.get(label, [])
value.append((box, prob))
objects[label] = value
|
Output:
{‘img’: [((126, 295, 530, 800), 0.5174897), ((166, 306, 497, 613), 0.510667), ((176, 484, 520, 656), 0.56631094), ((161, 304, 499, 613), 0.55209666), ((161, 306, 504, 613), 0.6020483), ((161, 306, 499, 613), 0.54256636), ((140, 305, 499, 800), 0.5012991), ((144, 305, 516, 800), 0.50028765), ((162, 305, 499, 642), 0.84315413), ((141, 306, 517, 800), 0.5257749), ((173, 433, 433, 610), 0.56347036)], ‘matchstick’: [((169, 633, 316, 800), 0.56465816), ((172, 633, 313, 800), 0.7206488), ((333, 639, 467, 800), 0.60068905), ((169, 633, 314, 800), 0.693922), ((172, 633, 314, 800), 0.70851576), ((167, 632, 314, 800), 0.6374499), ((172, 633, 316, 800), 0.5995729), ((169, 640, 307, 800), 0.67480534)], ‘guillotine’: [((149, 591, 341, 800), 0.59910816), ((149, 591, 338, 800), 0.7370558), ((332, 633, 469, 800), 0.5568006), ((142, 591, 341, 800), 0.6165994), ((332, 634, 468, 800), 0.63907826), ((332, 633, 468, 800), 0.57237893), ((142, 590, 321, 800), 0.6664309), ((331, 635, 467, 800), 0.5186203), ((332, 634, 467, 800), 0.58919555)], ‘water_tower’: [((144, 596, 488, 800), 0.50619787)], ‘barber_chair’: [((165, 465, 461, 576), 0.5565266)]}
As you can see it’s a dictionary where the label, ‘rocking chair’, is the key and we have a list of tuple which have bounding boxes and probabilities stored for this label as values.
Step 5: Show the Detected object on the Image
Look at the objects dictionary again if you don’t already know, we have multiple bounding boxes for a single label, so won’t there be a cluster in the image if directly show it on the image?
Therefore, we need to use the non_max_suppression method which we’ll solve this problem for us. But to use this function we need an array of bounding boxes and an array of probabilities, and it returns us an array of bounding boxes.
Python3
for label in objects.keys():
img_copy = img.copy()
boxes = np.array([pred[ 0 ] for pred in objects[label]])
proba = np.array([pred[ 1 ] for pred in objects[label]])
boxes = non_max_suppression(boxes, proba)
(startX, startY, endX, endY) = boxes[ 0 ]
cv2.rectangle(img_copy, (startX, startY),
(endX, endY), ( 0 , 255 , 0 ), 2 )
y = startY - 10 if startY - 10 > 10 else startY + 10
cv2.putText(img_copy, label, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.45 , ( 255 , 0 ), 2 )
cv2.imshow( "Regional proposal object detection" , img_copy)
cv2.waitKey( 0 )
|
Below is the complete implementation:
Python3
import numpy as np
import cv2
from tensorflow.keras.applications import Xception
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications import imagenet_utils
from tensorflow.keras.preprocessing.image import img_to_array
from imutils.object_detection import non_max_suppression
img = cv2.imread( '/content/img4.jpg' )
search = cv2.ximgproc.segmentation.createSelectiveSearchSegmentation()
search.setBaseImage(img)
search.switchToSelectiveSearchFast()
rects = search.process()
rois = []
boxes = []
(H, W) = img.shape[: 2 ]
for (x, y, w, h) in rects:
if w / float (W) < 0.1 or h / float (H) < 0.1 :
continue
roi = img[y:y + h, x:x + w]
roi = cv2.cvtColor(roi, cv2.COLOR_BGR2RGB)
roi = cv2.resize(roi, ( 299 , 299 ))
roi = img_to_array(roi)
roi = preprocess_input(roi)
rois.append(roi)
x1, y1, x2, y2 = x, y, x + w, y + h
boxes.append((x1, y1, x2, y2))
model = Xception(weights = 'imagenet' )
input_array = np.array(rois)
print ( "Input array shape is ;" , input_array.shape)
preds = model.predict(input_array)
preds = imagenet_utils.decode_predictions(preds, top = 1 )
objects = {}
for (i, pred) in enumerate (preds):
iD = pred[ 0 ][ 0 ]
label = pred[ 0 ][ 1 ]
prob = pred[ 0 ][ 2 ]
if prob > = 0.9 :
box = boxes[i]
value = objects.get(label, [])
value.append((box, prob))
objects[label] = value
for label in objects.keys():
img_copy = img.copy()
boxes = np.array([pred[ 0 ] for pred in objects[label]])
proba = np.array([pred[ 1 ] for pred in objects[label]])
boxes = non_max_suppression(boxes, proba)
(startX, startY, endX, endY) = boxes[ 0 ]
cv2.rectangle(img_copy, (startX, startY),
(endX, endY), ( 0 , 255 , 0 ), 2 )
y = startY - 10 if startY - 10 > 10 else startY + 10
cv2.putText(img_copy, label, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.45 , ( 255 , 0 ), 2 )
cv2.imshow( "Regional proposal object detection" , img_copy)
cv2.waitKey( 0 )
|
Output:
Last Updated :
29 Mar, 2023
Like Article
Save Article
Share your thoughts in the comments
Please Login to comment...