How to get detection score from OpenCV cascade classifier

The OpenCV cascade classifier makes it easy to train and test detection of faces and other objects in images. I have already written about how to train this classifier here. To get the objects detected during the testing stage, OpenCV provides this function:

void CascadeClassifier::detectMultiScale(
    const Mat& image,
    vector<Rect>& objects,
    double scaleFactor=1.1,
    int minNeighbors=3,
    int flags=0,
    Size minSize=Size(),
    Size maxSize=Size()
);

This returns a list of rectangles in the image where the classifier thinks it has detected the object. However, there seems to be no way to rank the detected locations. There is no detection or confidence weight or score associated with each rectangle to help you prioritize the detected locations. This is surprising since such a score should be present in the classifier, it is just that this function is not exposing that information.

Thankfully, there is an alternative to this function that returns the weight and reject level associated with each detected rectangle. Surprisingly, this function does not yet appear in the documentation. You can find it declared in modules/objdetect/include/opencv2/objdetect/objdetect.hpp as:

CV_WRAP virtual void detectMultiScale(
    const Mat& image,
    CV_OUT vector<Rect>& objects,
    vector<int>& rejectLevels,
    vector<double>& levelWeights,
    double scaleFactor=1.1,
    int minNeighbors=3,
    int flags=0,
    Size minSize=Size(),
    Size maxSize=Size(),
    bool outputRejectLevels=false
);

You can also hack the code a bit to get other types of scores with each detected location as demonstrated in this post.

Tried with: OpenCV 2.4.9 and Ubuntu 14.04

Tips for using OpenCV Cascade Classifier

My earlier post describes the basic steps to train the OpenCV Cascade Classifier. Unfortunately, the training process of opencv_traincascade is not for the newbie. It may take days to finish and the classifier might not work as you expected. Here is a list of tips that I found useful to train the classifier.

Rotation invariance

The cascade classifier is not rotation invariant. If your images can have the object in different rotations, then you have two options:

  1. You can generate a training set where the objects in the positive samples are rotated at different angles. You have to generate these yourself, opencv_createsamples will not do this for .vec file output.

  2. You strictly use positive samples where the object is in an upright or canonical orientation. For example, positive samples of faces where the face is always upright. You may have to write a program with some user interaction to generate these from your training samples. Later during the testing stage, the cascade classifier will only detect the object, if it appears in canonical position (in which it was trained). So, to get it to detect the object, you need to repeatedly rotate the image and try the classifier.

Rotation angle

While the cascade classifier is rotation invariant (see above), this does not mean that the object needs to be perfectly in upright or canonical orientation. The classifier should be able to detect object up to about 10 degrees of rotation. What this means is that:

  1. If you are using rotated positive samples, you do not need to try very small rotations (say 5 degrees). Instead, a larger rotation should be fine. For example, rotations in increments of 20 degrees.

  2. If you are rotating the image during the testing stage, then again no need to try small rotations. Same advice as earlier point.

Multithreading

If training is extremely slow, then build OpenCV with Intel TBB support. This can be done by installing the libtbb-dev package and then building OpenCV with the WITH_TBB option enabled. With this feature enabled (along with LBP), I found that opencv_traincascade which used to use 1 core, started to use all the available cores. Again, this can provide another 8x jump in training speed. With a speedup like this, I highly recommend to use this feature for everyone who is using an Intel CPU.

LBP features

If training is extremely slow, possibly taking days, then consider using the -featureType LBP option. By default, HAAR features are computed, which can be very slow. Local Binary Patterns can be almost an order of magnitude faster to compute. Its testing performance can match HAAR if you provide a good training set with lots of positive samples.

Related: Enable multithreading with TBB during cascade

Error: Can not get new positive sample

Problem

I was in the process of training a cascade classifier. I had exactly 354 positive samples and yet I got this error from opencv_traincascade when it entered stage 2:

$ opencv_traincascade -data car-classifier -vec pos-samples.vec -bg neg-image-filepaths.txt -precalcValBufSize 256 -precalcIdxBufSize 256 -numPos 354 -numNeg 4000 -nstages 20 -minhitrate 0.999 -maxfalsealarm 0.5 -w 50 -h 50 -nonsym -baseFormatSave 
PARAMETERS:
numPos: 354
numNeg: 4000
numStages: 20
stageType: BOOST
featureType: HAAR
sampleWidth: 50
sampleHeight: 50
boostType: GAB
minHitRate: 0.995
maxFalseAlarmRate: 0.5
weightTrimRate: 0.95
maxDepth: 1
maxWeakCount: 100
mode: BASIC

===== TRAINING 0-stage =====
<BEGIN
POS count : consumed   354 : 354
NEG count : acceptanceRatio    4000 : 1
Precalculation time: 100
+----+---------+---------+
|  N |    HR   |    FA   |
+----+---------+---------+
|   1|        1|        1|
+----+---------+---------+
|   2|        1|        1|
+----+---------+---------+
|   3|        1|        1|
+----+---------+---------+
|   4|        1|        1|
+----+---------+---------+
|   5| 0.997175|  0.40525|
+----+---------+---------+
END>
Training until now has taken 0 days 1 hours 39 minutes 31 seconds.

===== TRAINING 1-stage =====                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
<BEGIN
OpenCV Error: Bad argument (Can not get new positive sample. The most possible reason is insufficient count of samples in given vec-file.
) in get, file opencv-2.4.9/apps/traincascade/imagestorage.cpp, line 162
terminate called after throwing an instance of 'cv::Exception'
  what():  opencv-2.4.9/apps/traincascade/imagestorage.cpp:162: error: (-5) Can not get new positive sample. The most possible reason is insufficient count of samples in given vec-file.
 in function get

Solution

This problem seems to be very common. One possible solution is here. I took the hint from that and reduced the number of positive samples I provided as input to the program. Note that you need more positive samples to be present in the .vec file, just pass a smaller number to the program so it has some left over when it starts stage 1.

Here is the modified invocation I used:

$ opencv_traincascade -data car-classifier -vec pos-samples.vec -bg neg-image-filepaths.txt -precalcValBufSize 256 -precalcIdxBufSize 256 -numPos 200 -numNeg 2000 -nstages 20 -minhitrate 0.999 -maxfalsealarm 0.5 -w 50 -h 50 -nonsym -baseFormatSave

Tried with: OpenCV 2.4.9 and Ubuntu 14.04

How to train OpenCV cascade classifier

OpenCV ships with an application that can be used to train a cascade classifier. The steps to prepare your data and train the classifier can be quite elaborate. I have detailed the steps that I used below to train the classifier to identify an object (say car):

  • Two OpenCV programs: opencv_createsamples and opencv_traincascade will be used for this process. They should be installed along with your OpenCV. You can also compile them while compiling OpenCV from source. Make sure to enable the option BUILD_APPS in the CMake GUI. This is the option that builds these applications.

  • Prepare images that have one or more instances of the object of interest. These will be used to generate positive samples later. Also, prepare images that do not have the object of interest. These will be used to generate negative samples later.

  • It is mandatory to use opencv_createsamples to generate the positive samples for opencv_traincascade. The output of this program is a .vec file that is used as input to opencv_traincascade.

  • If you have image patches of the object and need to generate positive sample images by applying transformations (rotations) on this in 3 dimensions and then superimposing it on a background, you can do that using opencv_createsamples. See its documentation for details.

  • In my case, I already had one or more instances of the object captured with its actual background. So, what I instead had to do was to indicate the patch rectangles in each image where my object was located. You can do this by hand or by writing a simple OpenCV program to display image to you and you mark out the rectangles on the objects and stores these values in a text file. The format of this text file required by opencv_createsamples can be seen in its documentation.

  • To create the positive samples file, I invoked opencv_createsamples as:

$ opencv_createsamples -info obj-rects.txt -w 50 -h 50 -vec pos-samples.vec

Here, obj-rects.txt is a text file that has the information of the rectangles where the object is located in each image. See step above for details. The output of this program is stored in pos-samples.vec.

  • To view the positive samples that have been created by this program:
$ opencv_createsamples -vec pos-samples.vec -w 50 -h 50

Note that it switches to viewing mode when you only provide these three parameters and their values should match what you provided to create the positive samples.

  • Now we can train the cascade classifier. The details of its parameters can be seen in its documentation. I used this invocation:
$ opencv_traincascade -data obj-classifier -vec pos-samples.vec -bg neg-filepaths.txt -precalcValBufSize 2048 -precalcIdxBufSize 2048 -numPos 200 -numNeg 2000 -nstages 20 -minhitrate 0.999 -maxfalsealarm 0.5 -w 50 -h 50 -nonsym -baseFormatSave

obj-classifier is a directory where we are asking the classifier files to be stored. Note that this directory should already be created by you. pos-samples.vec is the file we generated in step above. neg-filepaths.txt is a file with list of paths to negative sample files. 2048 is the amount of MB of memory we are requesting the program to use. The more the memory, the faster the training. 200 is the number of positive samples in pos-samples.vec. This number is also reported by opencv_createsamples when it finishes its execution. 2000 is the number of negative sample image paths we have specified in neg-filepaths.txt. 20 is the number of stages in the classifier we wish. 50x50 is the size of object in these images. This should be same as what was specified with opencv_createsamples. 0.999 and 0.5 are self-explanatory. Details on all these parameters are found in the documentation.

Related: See my tips and other posts on using OpenCV Cascade Classifier.

Tried with: OpenCV 2.4.9 and Ubuntu 14.04