Train a Support Vector Machine to recognize facial features in C++

Let’s see how to train a model of support vector machine, save the trained model and test the model to check the percentage of its prediction accuracy using the OpenCV.

Data Organization:

Using imagenetscraper and autocrop , we collect data from the web, crop faces and resize them to smaller sizes in bulk. The collected data needs to be organized meaningfully so we can access it programmatically and manually. Use the below folder structure-

FFR_dataset/
|-- Age
|   |-- adult
|   |-- child
|   |-- old
|   |-- teen
|-- Emotion
|   |-- anger
|   |-- contempt
|   |-- happy
|   |-- neutral
|   |-- sad
|   |-- surprise
|-- Gender
    |-- female
    |-- male

We use the same directory names in the code to access them to train, save and predict the recognition results. A minimum of 50 images in each folder is required to train the models to get good prediction results. Training more images can improve the results but not recommended as it takes a lot of time to execute that and does not give significant improvements.

Implementation

By making use of the sample provided in the official opencv repo to train the SVM with HOG, train_HOG.cpp, we implement the c++ code to train, save and predict the facial features on an image with multiple faces.

There are three feature types- Age, Emotion and Gender. Four age groups, six emotions and two gender types. Hence an n-class classifier is implemented to recognize each feature on a face data.

Step #1: For each feature type i.e. (Age, Emotion or Gender) loop through ‘n’ run times.

// CTrainTestHOG::Run(int run_times)
for (auto ft : m_FeatureList) {
    DEBUGLW("\tFeature type=[%s]\n", ft.first.c_str());

    std::vector<float> predictionAccuracyList;
    predictionAccuracyList.reserve(run_times);

    for (int run = 0; run < run_times; ++run) {
        DEBUGLW("\t\tRun=[%d]\n", run);
        vector<Mat> trainData, predData;
        vector<int> trainLabels, predLabels;

        this->get_ft_dataset(ft.first, trainData, predData,
                                  trainLabels, predLabels);
        // ... train, predict and measure the SVM model.
    }
}

Step #2: In each run, iterate through the feature values in the feature type and get the images into a vector or array, i.e. get all the images from folders Gender->Male and Gender->Female.

// CTrainTestHOG::get_ft_dataset()
std::set<cv::String>& featureValueList = m_FeatureList.find(ft)->second;

for (auto fv : featureValueList) {
    DEBUGLW("\t\t\tFeature value=[%s]\n", fv.c_str());
    std::vector<cv::Mat> _trainData;
    std::vector<cv::Mat> _predData;
    std::vector<int> _trainLabels;
    std::vector<int> _predLabels;

    errCode = this->get_ftfv_dataset(ft, fv, _trainData, _predData,
                                       _trainLabels, _predLabels);
    if (errCode != EXIT_SUCCESS)
        break;
    trainData.insert(trainData.end(), _trainData.begin(), _trainData.end());
    predData.insert(predData.end(), _predData.begin(), _predData.end());
    trainLabels.insert(trainLabels.end(), _trainLabels.begin(), _trainLabels.end());
    predLabels.insert(predLabels.end(), _predLabels.begin(), _predLabels.end());
}

Step #3 to #6:

• Crop the images in the vector to the face rectangles and update the images vector with the new faces list.
• Perform any pre-processing tasks such as resizing to smaller size (64, 64) on each image in faces list.
• Shuffle the pre-processed face images in vector randomly to introduce random input data.
• Split the dataset into training(80%) and prediction(20%) data.

//  CTrainTestHOG::get_ftfv_dataset()
std::vector<cv::Mat> imgList;
this->get_images(folderName, imgList);
this->get_cropped_faces(imgList);
this->get_preprocessed_faces(imgList);

//-- return on empty img list to prevent seg fault
if (imgList.empty()) {
    errCode = EXIT_FAILURE;
    DEBUGLE("Error img list is empty!\n");
    break;
}
DEBUGLD("\t\t\timgList.size()=[%ld]\n", imgList.size());

std::random_shuffle(imgList.begin(), imgList.end());

// 80% for training
int trainPart = imgList.size() * 0.8;

// 20% for predicting
int predPart = imgList.size() - trainPart;
DEBUGLD("\t\t\ttrainPart=[%d], predPart=[%d]\n", trainPart, predPart);

trainData.reserve(trainPart);
predData.reserve(predPart);

ft_t::iterator ft_iter = m_FeatureList.find(ft);
fv_t::iterator fv_iter = ft_iter->second.find(fv);
int label = std::distance(ft_iter->second.begin(),  fv_iter);
DEBUGLD("\t\t\tlabel=[%d]\n", label);

int i = 0;
for (; i < trainPart; ++i) {
    trainData.push_back(imgList.at(i));
    trainLabels.push_back(label);
}
DEBUGLD("\t\t\ti=[%d], trainData.size()=[%ld],
        trainLabels.size()
        = [% ld]\n ", i, trainData.size(),
                       trainLabels.size());

for (; i < imgList.size(); ++i) {
    predData.push_back(imgList.at(i));
    predLabels.push_back(label);
}
DEBUGLD("\t\t\ti=[%d], predData.size()=[%ld],
        predLabels.size()
        = [% ld]\n ", i, predData.size(), predLabels.size());

Step #7 : Compute HOG for each image in training data.

// CTrainTestHOG::computeHOGs()
HOGDescriptor hog;
vector<Mat> hogMats;
vector<float> descriptors;
for (auto img : imgHogList) {
    hog.winSize = img.size() / 8 * 8;
    hog.compute(img, descriptors);
    cv::Mat descriptors_mat(Mat(descriptors).clone());
    hogMats.push_back(descriptors_mat);
}
imgHogList.swap(hogMats);

Step #8 : Convert the training data vector to opencv Mat object to train SVM.

//  CTrainTestHOG::convert_to_ml()
for (size_t i = 0; i < train_samples.size(); ++i) {
    CV_Assert(train_samples[i].cols == 1 || train_samples[i].rows == 1);
    if (train_samples[i].cols == 1) {
        cv::transpose(train_samples[i], tmp);
        tmp.copyTo(trainData.row((int)i));
    }
    else if (train_samples[i].rows == 1) {
        train_samples[i].copyTo(trainData.row((int)i));
    }
}

Step #9 : Pass the training data Mat object to svm train function along with a vector of labels for the training data.

//  CTrainTestHOG::Run()
trainLabels.resize(ml_train_data.rows);
// train svm
DEBUGLW("\t\tTraining SVM - begin\n");
m_pSVM->train(ml_train_data, ROW_SAMPLE, trainLabels);
DEBUGLW("\t\tTraining SVM - end\n");

Step #10: Save the trained model.

//-- step 10, CTrainTestHOG::Run()
cv::String svmModelFileName = cv::format("%s/cv4_svm_%s_model.xml",
                                         getenv(FFR_DATASET_PATH),
                                         ft.first.c_str());

m_pSVM->save(svmModelFileName.c_str());
DEBUGLW("\t\tSaved SVM model=[%s]\n",
        svmModelFileName.c_str());

Step #11: Predict the model by computing the HOG for each prediction image, convert the prediction dataset to opencv mat object and call svm predict with a vector of labels to store the result.

//-- step 11, CTrainTestHOG::Run()
// test the model
// compute HOG for each pre-processed face
errCode = this->computeHOGs(predData);
if (errCode != EXIT_SUCCESS) {
    DEBUGLE("Error in computing HOGs for the feature "
            "type=[%s]\n",
            ft.first.c_str());
    break;
}

// convert HOG feature vectors to SVM data
Mat ml_pred_data;
vector<int> resultLabels;
errCode = this->convert_to_ml(predData, ml_pred_data);
if (errCode != EXIT_SUCCESS) {
    DEBUGLE("Error in converting to ml for the "
            "feature type=[%s]\n",
            ft.first.c_str());
    break;
}
predLabels.resize(ml_pred_data.rows);
// resultLabels.resize(ml_pred_data.rows);
// test svm
DEBUGLW("\t\tTesting SVM - begin\n");
Mat responses_mat;
m_pSVM->predict(ml_pred_data, responses_mat);
for (size_t i = 0; i < ml_pred_data.rows; ++i) {
    resultLabels.push_back(responses_mat.at<int>(i));
}
DEBUGLW("\t\tTesting SVM - end\n");

Step #12 and #13: Calculate the percentage of its accuracy by comparing the expected prediction labels with the predicted labels.

//  CTrainTestHOG::Run()
// check the accuracy
float accuracy = 0.0f;
this->get_prediction_accuracy(predLabels, resultLabels, accuracy);
DEBUGLW("\t\tPrediction accuracy=[%lf]\n", accuracy);
predictionAccuracyList.push_back(accuracy);

//-- step 13, CTrainTestHOG::Run()
// check the mean accuracy of 'n' runs
float sum_of_accuracies = std::accumulate(
    predictionAccuracyList.begin(),
    predictionAccuracyList.end(), 0.0);
float mean_accuracy = sum_of_accuracies / predictionAccuracyList.size();
DEBUGLW("\t\tMean prediction accuracy=[%lf]\n",
        mean_accuracy);

Run the executable with below command line arguments.

./train_hog --test --in= --out= --show

Input:

Output:

Results log for HOG SVM using OpenCV 2.4
Results log for HOG SVM using OpenCV 4.0

Note: Due to long hair for all three people in the image it is detecting the gender as ‘female’ which is a false positive. In machine learning algorithms the false positives are always common given the input sample image has ambiguous features.