One of the irritating problems I encounter while working with CUDA programs is the GPU ID. This is the identifier used to associate an integer with a GPU on the system. This is just 0 if you have one GPU in the computer. But when dealing with a system having multiple GPUs, the GPU ID that is used by CUDA and GPU ID used by non-CUDA programs like nvidia-smi are different! CUDA tries to associate the fastest GPU with the lowest ID. Non-CUDA tools use the PCI Bus ID of the GPUs to give them a GPU ID.
One solution that I was using was to use cuda-smi that shows GPU information using CUDA GPU IDs.
There is a better solution: requesting CUDA to use the same PCI Bus ID enumeration order as used by non-CUDA programs. To do this set the
CUDA_DEVICE_ORDER environment variable to
PCI_BUS_ID in your shell. The default value of this variable is
FASTEST_FIRST. More info on this can be found here. Note that this is available only in CUDA 7 and later.
Reference: CUDA Environment Variables