One of the irritating problems I encounter while working with CUDA programs is the GPU ID. This is the identifier used to associate an integer with a GPU on the system. This is just 0 if you have one GPU in the computer. But when dealing with a system having multiple GPUs, the GPU ID that is used by CUDA and GPU ID used by non-CUDA programs like nvidia-smi are different! CUDA tries to associate the fastest GPU with the lowest ID. Non-CUDA tools use the PCI Bus ID of the GPUs to give them a GPU ID.
One solution that I was using was to use cuda-smi that shows GPU information using CUDA GPU IDs.
There is a better solution: requesting CUDA to use the same PCI Bus ID enumeration order as used by non-CUDA programs. To do this set the
CUDA_DEVICE_ORDER environment variable to
PCI_BUS_ID in your shell. The default value of this variable is
FASTEST_FIRST. More info on this can be found here. Note that this is available only in CUDA 7 and later.
I was compiling some old CUDA code with a recent version of the CUDA SDK. I got these errors on Thrust methods:
error: namespace thrust has no member max_element
error: namespace thrust has no member min_element
In recent versions of CUDA SDK, these Thrust methods have been moved to the
extrema.h header file. These errors will go away if you include the
thrust/extrema.h header file.
Tried with: CUDA SDK 7.5 and Ubuntu 14.04
PhysX is a 3D game physics engine provided by NVIDIA. They have released the source code of the engine on Github, though with restricted access. This library and its sample programs can be compiled from source easily.
Here are my notes on how to compile this library, run its sample tools and get started:
- You need a Linux computer (I use Ubuntu), with a fairly modern NVIDIA graphics card.
Make sure you have installed recent NVIDIA graphics drivers and a recent version of CUDA on it. Ensure that these are working correctly before trying PhysX.
Go to this webpage and jump through their hoops to get access to the PhysX Github page. Essentially, NVIDIA requires you to create a login with them and after that they give your Github login the access to their PhysX source code.
Once you have access to PhysX source code, clone its repository to your computer:
$ git clone https://github.com/NVIDIAGameWorks/PhysX-3.3.git
The documentation in the source code is outdated and is misleading. This is how the source code is laid out:
Source for source code,
Samples have small programs to try PhysX. Once you have compiled these Snippets and Samples, their binaries will be placed in
Each of the above three code directories has a
compiler/linux64 directory which holds the Makefiles to build them. There are four build profiles available:
debug. Just invoking
make builds all four versions. To build just the release versions, I did
make release in all the three code directories.
Once the library and its snippets and samples are built, you can try these programs from
Bin/linux64. For example, the
samples program allows you try many of the features of the engine in an interactive GUI.
Tried with: PhysX 3.3.4, NVIDIA GeForce GTX 750 Ti and Ubuntu 14.04
The easiest method is to visit the CUDA Downloads webpage and download the deb (network) file that matches your Ubuntu. Installing from it is as easy as:
$ sudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.deb
$ sudo apt-get update
$ sudo apt install cuda-7-5
This worked great on a notebook for me. However, on a desktop I started getting the infamous login loop problem.
Login loop problem: You start Ubuntu and you get the graphical login, but it is being displayed at an extremely low resolution (like
640x480 for example). You login, you see the desktop for a second, something fails and you are thrown back to the graphical login screen.
There are many solutions to the login loop problem. The only one which worked for me is described here.
That solution has many steps, not all of which I needed. I also ran into problems not listed there. Here is what worked for me:
- Purge all NVIDIA and CUDA packages:
$ sudo apt purge "nvidia*"
$ sudo apt purge "cuda*"
- Remove the X org configuration file, if it exists:
$ sudo rm /etc/X11/xorg.conf
- Reboot and make sure you are having a X desktop that is rendering fine at the correct resolution. If not, this guide cannot help you.
Visit the CUDA Downloads webpage and download the runfile (local) installation file. It is a 1+GB file and the download will take a while. This installer file contains both a NVIDIA graphics driver and the CUDA files.
Logout of your desktop and kill the X server:
$ sudo service lightdm stop
- Switch to a virtual terminal using
Ctrl + Alt + F1. Run the downloaded installer file. But, make sure it does not install its NVIDIA OpenGL libraries. This is the key to fixing the login loop problem! Choose Yes to everything except if it tries to create a
$ chmod +x cuda_7.5.18_linux
$ sudo ./cuda_7.5.18_linux --no-opengl-libs
At this point the NVIDIA driver installation failed saying that the
nouveau driver was active. It reported that it had disabled
nouveau, but required a restart.
I restarted the system, switched to virtual terminal, stopped X and ran the CUDA installer again, exactly as shown above. Again, the NVIDIA driver installation failed. This time it reported that the compiler was incompatible. I was using GCC 5.1 as my default compiler. I guess compiling NVIDIA driver kernel module needs something older. So, I switched back to GCC 4.8:
$ sudo update-alternatives --config gcc
- I restarted the installation and it finished successfully. I rebooted and was greeted by a desktop running at the correct resolution. CUDA programs worked fine too! 😄
Tried with: CUDA 7.5, NVIDIA driver 352.21, Ubuntu 14.04, Linux 3.19.0-28-generic and NVIDIA GTX 750 Ti
You have installed CUDA and try to compile a CUDA program using a CMake, which fails with this error:
$ cmake ..
CMake Error at /usr/share/cmake-2.8/Modules/FindCUDA.cmake:548 (message):
Call Stack (most recent call first):
FindCUDA.cmake is trying to find your CUDA installation directory and failing. I had installed CUDA 7.0 on this machine, which was in
/usr/local/cuda-7.0. However, CMake looks for
/usr/local/cuda. The CUDA installer is supposed to create a symbolic link
/usr/local/cuda pointing to that actual installation directory.
That symbolic link was not there on this computer. This can sometimes happen when you have two CUDA installations and remove one of them. The one removed takes out the symbolic link with it. I had CUDA 6.5 and CUDA 7.0 on this computer before I removed CUDA 6.5.
Anyway, we now know how to fix this:
$ sudo ln -s /usr/local/cuda-7.0 /usr/local/cuda
Pass the CUDA installation directory to the
CUDA_TOOLKIT_ROOT_DIR variable directly during the invocation of CMake:
$ cmake -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-7.0 ..
Tried with: CUDA 7.0 and Ubuntu 14.04
Vim has had syntax highlighting support for CUDA source files for a long time now. You can check this: open any
.cu file and try the command
:set filetype?. You will see that Vim knows that it is a CUDA source file. It applies the syntax highlighting from the file
So, what is the problem? The syntax highlighting done by Vim for CUDA is very very minimal. My CUDA source files look like pages of plain white text in Vim! Also, the comments in the
cuda.vim file that it was last updated in 2007. Now that is old!
There is an alternate
cuda.vim syntax file for Vim floating around on the web. You can get it, for example, from here.
I replaced the
cuda.vim that ships with Vim with this one and found it slightly better. But not by much. It still looks like lots of plain text.
The better solution for me was to just syntax highlight CUDA file as C++ file. This gave the best results, with the highest number of elements in the file being colored when compared to above two methods.
To do this, add this line to your
autocmd BufRead,BufNewFile *.cu set filetype=cpp
Tried with: Vim 7.4 and Ubuntu 14.04
target_include_directories is an useful CMake directive to specify the include directories for building a particular target, as described here. The FindCUDA module for CMake which handles CUDA compilation seems to completely ignore this directive. The include directories specified for the target are not passed to CUDA compilation by nvcc.
This will most commonly result in errors of the form:
someheader.h: No such file or directory.
This is a well known limitation of the CUDA module of CMake, as documented here. There seems to be no plan currently to support
target_include_directories for CUDA compilation.
The only solution is to switch to
include_directories to add these directories for all the targets in the
Tried with: CMake 22.214.171.124, CUDA 6.5 and Ubuntu 14.04
I was trying to install CUDA and got this error:
$ sudo apt-get install cuda
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
cuda : Depends: cuda-6-5 (= 6.5-14) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
On searching, a lot of folks had this error. Their errors were solved when they removed old NVIDIA packages they had installed. But, I did not have any NVIDIA package installed!
What were these broken packages I had on the system, I didn’t know! I followed the dependency chain by trying to install
cuda-6-5, which complained on another package and so on. In the end, I found that I had a lot of unnecessary packages, all of which had
:i386 at the end of their name.
The strangest part was that running
sudo apt-get autoremove had no effect on these packages, they were not removed. So, I manually removed all of them using
sudo apt-get remove.
When I tried to install CUDA after this, it worked fine! 🙂
Tried with: Ubuntu 14.04
Installing CUDA is becoming increasingly easier on Ubuntu. I think I keep hitting problems because I am usually updating from an older NVIDIA graphics driver or CUDA version. NVIDIA continues to be quite bad at providing error-free upgrades. Anyway, this is what worked for me:
- Do not try to install any of the NVIDIA drivers or CUDA packages that are in the Ubuntu repositories. I wasted a day with the errors these operations threw up!
Uninstall all CUDA packages and NVIDIA drivers you may have on your Ubuntu system.
Download the CUDA
.deb package for Ubuntu 14.04 from here. For me, it was a
.deb file just adds a CUDA repository maintained by NVIDIA. Install this
.deb file and update:
$ sudo gdebi cuda-repo-ubuntu1404_6.5-14_amd64.deb
$ sudo apt-get update
- Installing CUDA now is easy as this:
$ sudo apt-get install cuda
This is a big install, it will install everything including a
nvidia-340 driver that actually worked and NVIDIA NSight. After the install, reboot the computer. Your CUDA is ready for work now 🙂
Note: I tried this on two systems. On one, it installed without any problem. On the other, it gave an error of unmet dependencies. I have described here how I solved this problem.
Tried with: NVIDIA GeForce GTS 250 and NVIDIA GTX Titan
PyCUDA enables a Python program to pass data to and call CUDA kernels from a Python program. Getting it to install and work correctly on Ubuntu took a bit of work.
- I tried installing the
nvidia-343 drivers for Ubuntu. But, it turns out that only 340.x or earlier drivers support the GTS 250 I use for display (not for compute). So, I reverted back and installed
nvidia-331 drivers. Note that the
nvidia-331-updates driver will not work with CUDA either. Do not ask me why! 🙂
Installing the CUDA toolkit was easier, just install
PyCUDA is available in Ubuntu as a
python-pycuda package. But, that is the very old 2013.1.1 version. Instead I installed it from the Python Package Index (PyPI):
$ sudo pip install pycuda
The install script kept complaining about the absence of a
configure.py script, but it seemed to end with success.
Tried with: PyCUDA 2014.1, CUDA 5.5, NVIDIA 331 driver and Ubuntu 14.04