How to make CUDA and nvidia-smi use same GPU ID

One of the irritating problems I encounter while working with CUDA programs is the GPU ID. This is the identifier used to associate an integer with a GPU on the system. This is just 0 if you have one GPU in the computer. But when dealing with a system having multiple GPUs, the GPU ID that is used by CUDA and GPU ID used by non-CUDA programs like nvidia-smi are different! CUDA tries to associate the fastest GPU with the lowest ID. Non-CUDA tools use the PCI Bus ID of the GPUs to give them a GPU ID.

One solution that I was using was to use cuda-smi that shows GPU information using CUDA GPU IDs.

There is a better solution: requesting CUDA to use the same PCI Bus ID enumeration order as used by non-CUDA programs. To do this set the CUDA_​DEVICE_​ORDER environment variable to PCI_BUS_ID in your shell. The default value of this variable is FASTEST_FIRST. More info on this can be found here. Note that this is available only in CUDA 7 and later.

Reference: CUDA Environment Variables