No devices found error with OpenNI2

Problem

I installed OpenNI2 on a fresh Ubuntu system as described here. I plugged in a Primesense RD1.09 camera. I tried the default NIViewer app and it failed with this error:

openDevice failed:
DeviceOpen using default: no devices found

Solution

This error was quite a hard nut to crack. I could see that the Primesense camera was detected without errors in /var/log/syslog. I could see it being listed as ASUS when I run the lsusb command. And yet any OpenNI2 program would quit with the above error!

I could only solve this cause I remembered a similar error with SoftKinetic cameras. I tried the same solution as described here: creating a symbolic link to the libudev file.

After this, OpenNI2 programs began working immediately!

Tried with: OpenNI 2.2 x64, Primesense RD1.09 and Ubuntu 14.04

Advertisements

How to compute intrinsic camera matrix for a camera

The intrinsic camera matrix is useful in depth cameras to obtain the 3D position of any given pixel in the camera coordinate system. The pinhole camera model used for the intrinsic camera matrix is explained beautifully here.

The intrinsic camera matrix is of the form:

f_x s   x
0   f_y y
0   0   1

Here, f_x and f_y are the focal lengths of the camera in the X and Y directions. s is the axis skew and is usually 0. x and y are the X and Y dimensions of the image produced by the camera, measured from the center of the image. (So, they are half the length and width of the image.)

We typically know the dimensions of the image produced by the camera. What is typically not provided are the focal lengths. Instead camera manufacturers provide the field of view (FOV) angle in the horizontal and vertical directions.

Using the FOV angles, the focal lengths can be computed using trigonometry. For example, given the FOV a_x in the horizontal direction, the focal length f_x can be computed using:

f_x = x / tan(a_x / 2)

We divide the FOV by 2 because this angle spans the entire horizontal or vertical view.

As an example, consider the Primesense Carmine 1.09 depth camera. It produces a VGA (640×480) image. Its specifications state a horizontal FOV of 57.5 degrees and vertical FOV of 45 degrees.

Using the above information, we can compute its intrinsic camera matrix as:

583.2829786373293 0.0               320.0
0.0               579.4112549695428 240.0
0.0               0.0               1.0

How to reset Primesense firmware

I recently found that a Primesense camera was giving wrong depth values in its depth image. Suspecting an error in the firmware, I found a firmware downgrade that reverts to an old 579 version. This worked in resetting the firmware and the depth values were corrected.

  1. The steps to reset the firmware are similar to what is described here. You will need a Windows computer, remove OpenNI 1.x, remove Primesense driver and install OpenNI 2.x.

  2. Connect the Primesense camera. Download the DowngradeTo579.zip file and run the .bat file indicated in the readme.txt. That should reset the firmware to an older working version.

Tried with: Primesense RD1.09 and Windows 8.1 x64

How to log in OpenNI 2

I use OpenNI 2 to read depth and color (RGB) images from a Primesense camera. Sometimes when my program does not work as expected, I like to enable the logging that is built into OpenNI 2.

  • To change the logging settings, the file that needs to be edited is Redist/OpenNI.ini.

  • Log messages can be printed to the console by setting LogToConsole to 1. If you run your program after this is enabled, it prints log messages on the console.

  • Log messages can be written to a log file by setting LogToFile to 1. If you run your program after this is enabled, it prints log messages to files in a Log subdirectory. One log file is created for every invocation of your program. The log filename is a timestamp, for example: 2015_08_18__20_27_53_16182.log

  • Four levels of verbosity is available for log messages. In increasing verbosity these are: error (3), warning (2), info (1) and verbose (0).

Tried with: OpenNI 2.2, Primesense RD1.09 and Ubuntu 14.04

How to update Primesense firmware

My Primesense RD1.09 cameras were having lots of trouble working under Linux. They had no problems working under Windows.

One of the suggestions I found online was to update their firmware. Note that the firmware update can only be done from a Windows computer. I followed these steps to update:

  1. Remove all OpenNI 1.x or 2.x SDK installed on Windows.
  2. Remove the driver for Primesense camera as described here.
  3. Install the latest OpenNI 2.x SDK by downloading it from here.
  4. Plug in the Primesense camera and ensure that it is detected in Device Manager. Strangely my camera is detected as a Primesense 1.08x device! I have no idea why this happens.
  5. Get the firmware update software from here. There are two versions of firmware, which one to pick? See below.
  6. Unzip the firmware and run the EXE file. It should change some settings and show a SUCCESS in the console. If you get a FAILURE, then make sure you followed the OpenNI removal and driver removal steps above.

Which firmware version to pick? There are two versions: one for RD108 and another for RD109 devices. My camera says RD1.09 on the back, but shows up in Windows as a 1.08x device! Which version of firmware should I use? I tried both and here is my experience. When I used RD109 firmware, the camera would give this error on Linux:

One or more of the following nodes could not be enumerated:
Device: PrimeSense/SensorV2/5.1.0.41: The device is not connected!

I went back and tried the RD108 firmware and the camera worked under Linux after that 🙂

Tried with: OpenNI 2.2.0.33 and Windows 7 x64

OpenNI USB thread priority warning

Problem

I have a Primesense camera plugged into USB port. Running any program linked with the OpenNI library throws this warning:

Warning: USB events thread - failed to set priority. This might cause loss of data

Solution

OpenNI tries to set the USB async thread priority to critical. This works on Linux only with root privileges. For a normal user, the program will throw this warning. To get rid of this warning, run the program as root.

For more details see ThirdParty/PSCommon/XnLib/Source/Linux/XnLinuxUSB.cpp in OpenNI source code here.

Tried with: Primesense RD1.09, OpenNI 2.2 and Ubuntu 14.04

Primesense on Linux

My Primesense RD1.09 camera that causes many troubles
My Primesense RD1.09 camera that causes many troubles

I have a Primesense RD1.09 camera that works without any problem under Windows using OpenNI 2.2.0.33. Under Linux though, it is a world of pain!

Investigation

After a lot of debugging, I have discovered that most of the problems arise if you have a computer which has USB 3.0 controllers or hubs. It does not matter if you plug in the Primesense into a USB 3.0 or USB 2.0 port. Just the existence of USB 3.0 on your computer means that the Linux kernel behaves differently.

After endless hours of plugging and unplugging into USB ports and poring over the errors in /var/log/syslog and the behavior of lsusb on 3 different computers, I believe it is the USB subsystem of Linux that is causing the problem. Linux kernel uses the XHCI module and on computers which have USB 3.0 controllers, it seems to be causing problems. For an excellent example see this scanner problem.

Additionally, if you have played around with different versions of OpenNI, that can also add another layer of bugs. Make sure you apt-get remove --purge when you remove OpenNI 1.x. Also make sure you manually remove the files installed by OpenNI 2.x (which does not have an uninstaller).

Symptoms

Check if you see of these symptoms when using Primesense in Linux:

  • You plug in Primesense and type lsusb. It should enumerate the USB devices in a second. If it takes longer, you have a computer with USB 3.0 ports and the buggy Linux kernel USB modules are having trouble with it.

  • Primesense is detected in lsusb and it appears as ASUS. But, running any sample viewer program returns in error.

  • Primesense is detected only half the times it is plugged in.

  • When the camera is detected, I run a program to read depth and color streams. Half the time, the color stream does not provide any frames!

  • Even when both depth and color frames are provided, the color frames have lower FPS and even freeze sometimes!

Solution

Both of these solutions worked for me:

  • On computer with USB 3.0, go to BIOS setup and disable Intel xHCI and EHCI. If your BIOS does not have these options, you are out of luck! 😦

  • Switch to an older computer which has only USB 2.0 controllers and ports.

Primesense worked flawlessly with the OpenNI 1.x packages (from Ubuntu) on the computers after I tried the above solutions! 🙂

USB interface error with OpenNI and Primesense

Problem

I wanted to use a Primesense RD1.09 depth camera on my Ubuntu computer. I installed the OpenNI packages available in Ubuntu as described here. I plugged in the camera to a USB 2.0 port and tried the default OpenNI viewer and got this error:

$ Sample-NiSimpleViewer 
Open failed: Failed to set USB interface!

Solution

The camera seemed to be detected since it appears in lsusb listing. The solutions provided online did not work. In the end, I removed the installed OpenNI packages and installed the latest OpenNI2 package as described here. This worked! 🙂

Tried with: Primesense RD1.09 and Ubuntu 14.04

How to install and use OpenNI2

I use OpenNI to read from a Primesense RD1.09 depth camera. Ubuntu has OpenNI library packages in its archive, but this is the older OpenNI 1.x. I find its API and device support lacking and so prefer to use OpenNI2. Also, there is no need to build the newer OpenNI2 from source since pre-built packages are available.

Installing and using OpenNI2 with Primesense is easy:

  • Download: The pre-built SDK file for your Linux can be downloaded from the OpenNI homepage. It is available for x86, x64 and ARM. I downloaded the x64 SDK file and unzipped it.

  • OpenNI directory: Your OpenNI code and programs will look for header and library files from this unzipped directory. So, move it to a suitable location. I like to keep it in ~/lib

  • Install: Run the installer from the unzipped directory:

$ sudo ./install.sh

All this does is to place the UDEV rules for the Primesense camera in /etc/udev/rules.d. Your camera is now ready to be used!

  • Camera detection: Plug in your depth camera to a USB port on your computer. Check if the camera is detected using lsusb command. My camera is listed as ASUS in this listing when it is detected.

  • Simple viewer: Run the simple viewer provided in OpenNI. You should be able to view depth and color (RGB) images now:

$ cd Tools
$ ./NiViewer
  • Device error: If NiViewer failed with a no devices found error, then a solution I discovered is described here.

  • Color image error: If NiViewer only shows depth image, but no color image: this sometimes happens. Unplug and plug in the camera and try again.

  • Header and libraries: You will find a OpenNIDevEnvironment file containing the paths for two environment variables: OPENNI2_INCLUDE and OPENNI2_REDIST. Use these include and library paths when you build your own C++ code that uses OpenNI. To link with OpenNI libraries add the -lOpenNI2 directive.

  • Sample C++ project: If you need help starting off, I have a CMake project with code for reading from Primesense camera here.

Tried with: OpenNI 2.2.0.33, Primesense RD1.09 and Ubuntu 14.04

Depth to color image registration fails in OpenNI

Problem

I wrote C++ code to read depth and color images from a Primesense camera using OpenNI 2, running on Windows. After initializing OpenNI and opening the Primesense device, I check if depth-to-color image registration mode is supported by calling psense_device.isImageRegistrationModeSupported(openni::IMAGE_REGISTRATION_DEPTH_TO_COLOR). This returns openni::STATUS_OK. So, next I set this image registration mode by calling psense_device.setImageRegistrationMode(openni::IMAGE_REGISTRATION_DEPTH_TO_COLOR). However, this fails!

Solution

I could not actually figure out why OpenNI reports that the device supports this mode, but fails setting it. I suspected that maybe the camera needs to see depth and color video streams being created before this mode is set. So, I set this mode as late as possible, after the depth-color streams are opened and configured. That seemed to work!

Tried with: Primesense camera, OpenNI 2, Visual Studio 2013 and Windows 7 Professional 64-bit