target_include_directories does not work with CUDA

Problem

target_include_directories is an useful CMake directive to specify the include directories for building a particular target, as described here. The FindCUDA module for CMake which handles CUDA compilation seems to completely ignore this directive. The include directories specified for the target are not passed to CUDA compilation by nvcc.

This will most commonly result in errors of the form: someheader.h: No such file or directory.

Solution

This is a well known limitation of the CUDA module of CMake, as documented here. There seems to be no plan currently to support target_include_directories for CUDA compilation.

The only solution is to switch to include_directories to add these directories for all the targets in the CMakeLists.txt file.

Tried with: CMake 2.8.12.2, CUDA 6.5 and Ubuntu 14.04

Advertisements

CUDA installation error of unmet dependencies

I was trying to install CUDA and got this error:

$ sudo apt-get install cuda
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cuda : Depends: cuda-6-5 (= 6.5-14) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

On searching, a lot of folks had this error. Their errors were solved when they removed old NVIDIA packages they had installed. But, I did not have any NVIDIA package installed!

What were these broken packages I had on the system, I didn’t know! I followed the dependency chain by trying to install cuda-6-5, which complained on another package and so on. In the end, I found that I had a lot of unnecessary packages, all of which had :i386 at the end of their name.

The strangest part was that running sudo apt-get autoremove had no effect on these packages, they were not removed. So, I manually removed all of them using sudo apt-get remove.

When I tried to install CUDA after this, it worked fine! 🙂

Tried with: Ubuntu 14.04

How to install CUDA 6.5 on Ubuntu 14.04

Installing CUDA is becoming increasingly easier on Ubuntu. I think I keep hitting problems because I am usually updating from an older NVIDIA graphics driver or CUDA version. NVIDIA continues to be quite bad at providing error-free upgrades. Anyway, this is what worked for me:

  • Do not try to install any of the NVIDIA drivers or CUDA packages that are in the Ubuntu repositories. I wasted a day with the errors these operations threw up!

  • Uninstall all CUDA packages and NVIDIA drivers you may have on your Ubuntu system.

  • Download the CUDA .deb package for Ubuntu 14.04 from here. For me, it was a cuda-repo-ubuntu1404_6.5-14_amd64.deb file.

  • The .deb file just adds a CUDA repository maintained by NVIDIA. Install this .deb file and update:

$ sudo gdebi cuda-repo-ubuntu1404_6.5-14_amd64.deb
$ sudo apt-get update
  • Installing CUDA now is easy as this:
$ sudo apt-get install cuda

This is a big install, it will install everything including a nvidia-340 driver that actually worked and NVIDIA NSight. After the install, reboot the computer. Your CUDA is ready for work now 🙂

Note: I tried this on two systems. On one, it installed without any problem. On the other, it gave an error of unmet dependencies. I have described here how I solved this problem.

Tried with: NVIDIA GeForce GTS 250 and NVIDIA GTX Titan

How to install PyCUDA

PyCUDA enables a Python program to pass data to and call CUDA kernels from a Python program. Getting it to install and work correctly on Ubuntu took a bit of work.

  • I tried installing the nvidia-343 drivers for Ubuntu. But, it turns out that only 340.x or earlier drivers support the GTS 250 I use for display (not for compute). So, I reverted back and installed nvidia-331 drivers. Note that the nvidia-331-updates driver will not work with CUDA either. Do not ask me why! 🙂

  • Installing the CUDA toolkit was easier, just install nvidia-cuda-toolkit package.

  • PyCUDA is available in Ubuntu as a python-pycuda package. But, that is the very old 2013.1.1 version. Instead I installed it from the Python Package Index (PyPI):

$ sudo pip install pycuda

The install script kept complaining about the absence of a configure.py script, but it seemed to end with success.

Tried with: PyCUDA 2014.1, CUDA 5.5, NVIDIA 331 driver and Ubuntu 14.04

CUDA error: gcc 4.9 and up are not supported

Problem

I have seen this error while compiling a CUDA program. I have also seen this error on running a PyCUDA program:

#error -- unsupported GNU version! gcc 4.9 and up are not supported!

Solution

I was using GCC 4.9.2 on this system, but I did not know why CUDA had a problem with these newer versions. Turns out that it does not have any problem, this warning just needs to be choked in the relevant header file.

For CUDA, this header file was /usr/local/cuda/include/host_config.h. For PyCUDA, this header file was /usr/include/host_config.h.

This diff should fix this error:

--- host_config.h.bkp   2015-02-24 09:53:55.232620612 +0800
+++ host_config.h   2015-02-24 10:24:01.428654521 +0800
@@ -77,11 +77,11 @@

 #if defined(__GNUC__)

-#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 8)
+#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9)

 #error -- unsupported GNU version! gcc 4.9 and up are not supported!

-#endif /* __GNUC__> 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 8) */
+#endif /* __GNUC__> 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9) */

 #endif /* __GNUC__ */

Tried with: PyCUDA 2014.1, CUDA 6.5, GCC 4.9.2 and Ubuntu 14.04

PyCUDA error: cuInit failed

Problem

After installing PyCUDA, I tried to run a sample PyCUDA program and got this error:

$ python hello_cuda.py
Traceback (most recent call last):
  File "hello_cuda.py", line 1, in <module>
    import pycuda.autoinit
  File "/usr/local/lib/python2.7/dist-packages/pycuda/autoinit.py", line 4, in <module>
    cuda.init()
pycuda._driver.Error: cuInit failed: unknown

Solution

The error refers to line 4 in /usr/local/lib/python2.7/dist-packages/pycuda/autoinit.py. The lines relevant to that error in the file are:

import pycuda.driver as cuda
cuda.init()

So, CUDA initialization is failing for some reason. After a bit of Googling, I found that this means that the necessary (CUDA? NVIDIA?) modules were not being loaded into the kernel. NVIDIA provides a modprobe package that can auto-load modules into the kernel as needed.

I installed this package:

$ sudo apt-get install nvidia-modprobe

CUDA was able to initialized immediately after this package was installed, no reboot was needed.

Tried with: PyCUDA 2014.1, CUDA 5.5, NVIDIA 331 drivers, GTX Titan and Ubuntu 14.04

CMake error on CUDA toolkit

Problem

I installed CUDA using this command:

$ sudo apt install nvidia-cuda-dev

I tried to compile a CUDA program using CMake and got this error:

CMake Error at /usr/share/cmake/Modules/FindCUDA.cmake:488 (message):
  Specify CUDA_TOOLKIT_ROOT_DIR
Call Stack (most recent call first):
  CMakeLists.txt:20 (find_package)

Solution

Turns out that nvidia-cuda-dev does not install the CUDA toolkit or SDK. Though it is named like a development package in Linux typically is, it is not.

To install the CUDA toolkit use:

$ sudo apt install nvidia-cuda-toolkit

The CMake compile error went away after this 🙂

Tried with: CUDA 5.5, CMake 2.8.12.2 and Ubuntu 14.04

How to install CUDA from Ubuntu repositories

The Ubuntu repositories now host CUDA packages. These may not be latest and greatest, but I find them to work well with the NVIDIA drivers from the same place. Also, they are very easy to install compared to the packages from the NVIDIA website.

  • First, install the latest NVIDIA driver available from the repositories. NVIDIA driver package is named as nvidia-xyz, where xyz is the version. Pick the largest number version that is available from the repositories. For example:
$ sudo apt install nvidia-331

The installation process compiles the driver for your particular Linux kernel and deploys that module. Restart the computer once the install is done.

You must be able to see a NVIDIA module when you list the kernel modules. For example, on my computer:

$ dkms status
nvidia-331, 331.38, 3.13.0-24-generic, x86_64: installed
  • Now you are ready to install CUDA. This is really easy since installing the package nvidia-cuda-toolkit will pull in all the hundred other CUDA packages and tools that are needed:
$ sudo apt install nvidia-cuda-toolkit

That is it, enjoy your CUDA programming! 🙂

Tried with: Linux kernel 3.13.0-24-generic, CUDA 5.5, NVIDIA driver 331, NVIDIA GTX Titan and Ubuntu 14.04

How to ignore unrecognized GCC pragma warning of CUDA

Problem

You have a GCC pragma in your CUDA code to ignore a specific warning produced by your code. While the GCC compiler uses this for the C++ section of the code, the CUDA compiler frontend complains about it with a warning:

foo.cu(44): warning: unrecognized GCC pragma

Solution

To request the frontend of the CUDA compiler to ignore or hide this warning, pass this option during compilation:

-Xcudafe "--diag_suppress=unrecognized_gcc_pragma"

The GCC pragma warning should be hidden now.

Tried with: CUDA 5.5, GCC 4.8 and Ubuntu 12.04

NVProf

Output of nvprof
Output of nvprof

If you work with CUDA programs, you will use the Visual Profiler regularly. Another tool that can be useful is the commandline profiler, named nvprof. This does not have as many features of the Visual Profiler, but is very easy and quick to use.

Pass the name of your program to nvprof:

$ nvprof ./my-cuda-program

nvprof runs the program and gives a summary of results that is similar to the default output in Visual Profiler. It shows the kernels sorted in decreasing order of total execution times. The columns show the percentage of execution time, the actual time, the number of calls, the average-min-max of a single call for every kernel.

Tried with: NVProf 5.5, CUDA 5.5 and Ubuntu 12.04