How to show images in split window in OpenCV

A typical scenario in computer vision is to show multiple images in different parts of a single window, like a split window. However, OpenCV can only display a single cv::Mat in a single window using cv::imshow(). So, to achieve what we want, we create a single cv::Mat whose size is equal to the window. We copy our multiple images into this mat at the locations we want and just display the big mat.

(Note that this is far easier than the ROI regions you would need to use with IplImage.)

This sample illustrates showing two 640x480 images side-by-side in a single window:

// 640x480 images
cv::Mat mat_1;
cv::Mat mat_2;

// Create 1280x480 mat for window
cv::Mat win_mat(cv::Size(1280, 480), CV_8UC3);

// Copy small images into big mat
mat_1.copyTo(win_mat(cv::Rect(  0, 0, 640, 480)));
mat_2.copyTo(win_mat(cv::Rect(640, 0, 640, 480)));

// Display big mat
cv::imshow("Images", win_mat);

Tried with: OpenCV 2.4.8 and Ubuntu 14.04

Advertisements

Depth and type of matrix in OpenCV

cv::Mat is the most fundamental datatype used in OpenCV. It can be used to store 2D images with 1-4 channels of data.

When your code receives a cv::Mat from an external library or code, the most common question you have is what is the data type of the elements of this image? There seem to be two methods in the cv::Mat class that answer this: depth() and type().

Depth

Depth is the more fundamental of the two queries. It is the data type of each individual element in the image data. It can take one of these 8 values:

#define CV_8U   0
#define CV_8S   1
#define CV_16U  2
#define CV_16S  3
#define CV_32S  4
#define CV_32F  5
#define CV_64F  6
#define CV_USRTYPE1 7

For example, an image with 2 channels of unsigned short data will have depth of CV_16U.

Type

Type combines the data type of the elements along with the number of channels in the image. Since images in OpenCV can have 1-4 channels, it can take one of these 28 values:

#define CV_8UC1 CV_MAKETYPE(CV_8U,1)
#define CV_8UC2 CV_MAKETYPE(CV_8U,2)
#define CV_8UC3 CV_MAKETYPE(CV_8U,3)
#define CV_8UC4 CV_MAKETYPE(CV_8U,4)

#define CV_8SC1 CV_MAKETYPE(CV_8S,1)
#define CV_8SC2 CV_MAKETYPE(CV_8S,2)
#define CV_8SC3 CV_MAKETYPE(CV_8S,3)
#define CV_8SC4 CV_MAKETYPE(CV_8S,4)

#define CV_16UC1 CV_MAKETYPE(CV_16U,1)
#define CV_16UC2 CV_MAKETYPE(CV_16U,2)
#define CV_16UC3 CV_MAKETYPE(CV_16U,3)
#define CV_16UC4 CV_MAKETYPE(CV_16U,4)

#define CV_16SC1 CV_MAKETYPE(CV_16S,1)
#define CV_16SC2 CV_MAKETYPE(CV_16S,2)
#define CV_16SC3 CV_MAKETYPE(CV_16S,3)
#define CV_16SC4 CV_MAKETYPE(CV_16S,4)

#define CV_32SC1 CV_MAKETYPE(CV_32S,1)
#define CV_32SC2 CV_MAKETYPE(CV_32S,2)
#define CV_32SC3 CV_MAKETYPE(CV_32S,3)
#define CV_32SC4 CV_MAKETYPE(CV_32S,4)

#define CV_32FC1 CV_MAKETYPE(CV_32F,1)
#define CV_32FC2 CV_MAKETYPE(CV_32F,2)
#define CV_32FC3 CV_MAKETYPE(CV_32F,3)
#define CV_32FC4 CV_MAKETYPE(CV_32F,4)

#define CV_64FC1 CV_MAKETYPE(CV_64F,1)
#define CV_64FC2 CV_MAKETYPE(CV_64F,2)
#define CV_64FC3 CV_MAKETYPE(CV_64F,3)
#define CV_64FC4 CV_MAKETYPE(CV_64F,4)

For example, an image with 2 channels of unsigned short data will have type of CV_16UC2.

Here are two simple functions that are handy to get the depth and type of image as a string:

Reference: modules/core/include/opencv2/core/hal/interface.h in OpenCV source code

How to resize or rescale in OpenCV

An image cannot be resized or rescaled inplace in OpenCV. You will need to create another image with the new size or scale and apply a resize operation.

Resizing or rescaling a Mat is somewhat easier than dealing with a IplImage. The code below illustrates these operations on both data types:

Mat in OpenCV

Mat is the basic C++ structure used in OpenCV to hold two and higher dimensional data. IplImage is the deprecated older C structure that was used for the same purpose earlier. The advantage of using Mat over IplImage is that it behaves like a proper C++ object, so it is easier to construct and is automatically destroyed at end of its scope.

  • Documentation about the Mat class can be found here. The declaration of the Mat class is in the modules/core/include/opencv2/core/mat.hpp file, as can be seen here.

  • The basic structure of Mat is simple:

class Mat
{
public:
    Mat();
    Mat(int rows, int cols, int type);
    Mat(Size size, int type);
    Mat(const Mat& m);
    Mat(int rows, int cols, int type, void* data, size_t step=AUTO_STEP);
    Mat(Size size, int type, void* data, size_t step=AUTO_STEP);

    int flags;
    int dims;
    int rows, cols;
    uchar* data;
};
  • To construct a Mat from existing array of data, without making a copy to its own buffer:
double darr[640*480];
cv::Mat dmat(480, 640, CV_64F, darr);
  • To construct a Mat from existing array of data, by making a copy to its own buffer:
double darr[640*480];
cv::Mat dmat(480, 640, CV_64F, darr);
cv::Mat cmat;
dmat.clone(cmat); // cmat has a separate copy of data now
  • The data type of the elements of the image and the number of channels can be found using the depth and type methods as described here.

  • To set or get directly from the buffer of Mat:

// Set pixel 10 of CV_8UC3 mat to grey
cmat.data[3*10    ] = 40;
cmat.data[3*10 + 1] = 40;
cmat.data[3*10 + 2] = 40;
  • To convert between Mat and IplImage:
// To make Mat from IplImage*
// IplImage* img = cvCreateImage( ... );
cv::Mat mat(img);

// To make IplImage* from Mat
IplImage* img = new IplImage(mat);

IplImage in OpenCV

You might run into the IplImage structure while looking at OpenCV code. It is an old structure that is present from the days when OpenCV was called Intel Processing Library, hence IPL.

The different members of IplImage and their descriptions can be found here. The declaration of IplImage can be found in modules/core/include/opencv2/core/types_c.h of the OpenCV source code, as can be seen here.

The basic structure of IplImage is:

#define IPL_DEPTH_1U     1
#define IPL_DEPTH_8U     8
#define IPL_DEPTH_16U   16
#define IPL_DEPTH_32F   32
#define IPL_DEPTH_SIGN 0x80000000
#define IPL_DEPTH_8S  (IPL_DEPTH_SIGN| 8)
#define IPL_DEPTH_16S (IPL_DEPTH_SIGN|16)
#define IPL_DEPTH_32S (IPL_DEPTH_SIGN|32)

struct IplImage
{
    int depth; // Datatype of pixel (see above)
    int nChannels; // 1, 2, 3 or 4
    int width;
    int height;
    char* imageData;
    int widthStep;
};

Constructing it is easy:

IplImage* img = cvCreateImage(cvSize(200, 200), IPL_DEPTH_32F, 1); // 1-channel float image

The modern Mat structure in OpenCV is preferable to using IplImage. IplImage can be converted to Mat as described here.

Relational operators and Mat in OpenCV

Mat is a common structure used in OpenCV to store two and higher dimensional data. It is a derivation of the internal Array structure. A relational operator can applied between two Mat objects or a Mat object and a scalar, and the result is a 8-bit single-channel Mat.

Any of these operations is nothing but a call to compare function. This results in a per-element comparison of the Mat. The key information to remember is that for every element whose result is true, the value in the result Mat is set to 0xFF, that is 255. If result is false, it is set to 0. This behavior is so that the resulting Mat can be used as a mask image if needed for other operations.

The operators that are supported are equal, not-equal, greater, greater-or-equal, lesser and lesser-or-equal.

Example usage of such operations:

cv::Mat m0, m1;
cv::Mat m2(m0 == 0);
m2 = m0 > m1;
m2 = m1 <= m0;
m2 = m1 != 1;

How to read and display image in OpenCV

OpenCV can be used to easily read an image file and display it in a window. The core and highgui modules are needed to achieve this. The image is read into a cv::Mat object. cv::Mat is the modern replacement for the old IplImage object, which came from Intel.

#include <opencv2/highgui.hpp>

// Read image from file
cv::Mat img = cv::imread("foobar.jpg");

// Create a window
cv::namedWindow("foobar");

// Display image in window
cv::imshow("foobar", img);

// Wait for user to press a key in window
cv::waitKey(0);

To be able to compile this code, link with the core and highgui modules using -lopencv_core -lopencv_highgui.

The legacy method to achieve the same using IplImage is:

#include <opencv2/opencv.hpp>

// Read image from file
IplImage* img = cvLoadImage("foobar.jpg");

// Display image in window
cvLoadImage("foobar", img);

// Wait for user to press a key in window
cvWaitKey(0);

Tried with: Ubuntu 12.04 LTS

OpenCV error constructing Mat from IplImage

Problem

Tried to compile some code that uses OpenCV and got this error:

foo.cpp:268:27: error: no matching function for call to ‘cv::Mat::Mat(IplImage*&)’

The code at that error line tries to construct a cv::Mat object from IplImage:

void boohoo(IplImage* i)
{
    // ...
    abracadabra(cv::Mat(i));
    // ...
}

Solution

It turns out this code worked in older versions of OpenCV. But with recent versions of OpenCV, this cv::Mat constructor is no longer present.

Instead, the conversion has to be performed using a cv::cvarrToMat function:

void boohoo(IplImage* i)
{
    // ...
    abracadabra(cv::cvarrToMat(i));
    // ...
}

Tried with: OpenCV 2.4.9 and Ubuntu 12.04 LTS