nvidia-smi (NVIDIA System Management Interface) is a tool to query, monitor and configure NVIDIA GPUs. It ships with and is installed along with the NVIDIA driver and it is tied to that specific driver version. It is a tool written using the NVIDIA Management Library (NVML).
- To query the usage of all your GPUs:
I use this default invocation to check:
- Version of driver.
- Names of the GPUs.
- Index of the GPUs, based on PCI Bus Order. This is different from the CUDA order.
- Amount of memory each of the GPUs has.
- Whether persistence mode is enabled on each of the GPUs
- Utilization of each of the GPUs (if I’m running something on them).
- List of processes executing on the GPUs.
- To query the configuration parameters of all the GPUs:
$ nvidia-smi -q
I use this to check:
- Default clocks (listed under Default Application Clocks).
- Current clocks (listed under Application Clocks).
- To query the configuration parameters of a particular GPU, use its index:
$ nvidia-smi -q -i 0