Nvidia-smi
lastmod 2025-12-05created 2022-10-0511 edits
NVIDIA System Management Interface
Checking graphics cards
The output looks like this:
GPU 0: NVIDIA GeForce RTX 3070 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
GPU 1: NVIDIA GeForce RTX 3070 Ti (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
Changing persistence mode
Persistence Mode is the term for a user-settable driver property that keeps a target GPU initialized even when no clients are connected to it. This solution is near end-of-life and will eventually be deprecated in favor of the persistence daemon. Persistence mode can be set using nvidia-smi or programmatically via the NVML API.
- Persistence mode always preloads the nvidia driver even when the GPU is not in use, which helps programs start faster
- Power persistence option. When ON, it saves latency when using the GPU but wastes power.
To enable persistence mode using nvidia-smi (as root):
-
-pm,--persistence-mode= - Set persistence mode: 0/DISABLED, 1/ENABLED
How to check:
The output looks like this:
==============NVSMI LOG==============
Timestamp : ----
Driver Version : ----
Attached GPUs : ----
GPU 0000:01:00.0
Product Name : ----
Display Mode : ----
Display Active : ----
Persistence Mode : Enabled
Accounting Mode : ----
...
Supported Environments:
- Drivers: All shipping driver versions
- OSes: All standard driver-supported Linux platforms
- GPUs: All shipping Tesla, Quadro and GRID products
ECC settings
-
-e,--ecc-config= - Toggle ECC support: 0/DISABLED, 1/ENABLED
Setting compute mode
-
-c,--compute-mode= - Set MODE for compute applications
- 0/DEFAULT
- 1/EXCLUSIVE_THREAD (DEPRECATED),
- 2/PROHIBITED
- 3/EXCLUSIVE_PROCESS
Switch GPU Virtualization Mode
-
-vm,--virt-mode= - Switch GPU Virtualization Mode:
- Sets GPU virtualization mode to 3/VGPU or 4/VSGA Virtualization mode of a GPU can only be set when it is running on a hypervisor.
Trigger reset of the GPU
-
-r,--gpu-reset - Trigger reset of the GPU.
Can be used to reset the GPU HW state, which would otherwise require a machine reboot. Typically useful when a double-bit ECC error has occurred.
Description of output fields
- (NVIDIA) Description of nvidia-smi fields
- (CUDA) How to check and use NVIDIA-SMI
- ITMAYA - How much can you check with nvidia-smi?
- (Deep learning) How to read NVIDIA GPU info (nvidia-smi)
Running plain nvidia-smi with no options prints the following:
- Driver Version 470.103.01: The driver version of the nvidia GPU currently installed and in use.
- CUDA Version 11.4: The CUDA version that is compatible with the driver currently in use. This is only a recommendation; it does not refer to the CUDA version currently installed and in use.
- GPU Fan 0 48%: The number (0~7) of the installed GPU, and the fan performance of a GPU with a fan, shown as a %. (Tesla-series GPUs have no fan, so this is shown as N/A (Not Available))
- Name NVIDA RTX A6000: The GPU model name.
- Temp 70C: The current GPU temperature in Celsius. If the GPU temperature exceeds a certain value, performance drops. Usually between 70 and 80 degrees is an appropriate temperature.
- Perf P2: Short for Performance; ranges from P0 to P12. The smaller the number, the better the GPU performance state.
- Persistence-M Off: Persistence-Mode, which has an on-mode and an off-mode. The default mode is off; in on-mode you can set a power limit. In other words, you can decide how long power is sustained. The advantage is that it saves latency when using the GPU, but the disadvantage is that it wastes more power.
- Pwr:Usage/Cap 192W /300W: The GPU's current power usage and maximum capacity.
- Bus-ID 00000000:1B:00.0: The Bus-ID assigned to a PCI slot on each main board. With it you can match the GPU number in use to the main board's PCI slot. (Important!)
- Disp.A off: Linux comes in a Desktop version and a Server version. For the Desktop version, it is better to output the display through the GPU card. It is off by default, but for the GPU whose output port has a monitor connected, it is on, meaning output is going to that GPU card.
- Memory-Usage 21787MiB / 48682MiB: The GPU memory currently in use and the total GPU memory; memory size differs from GPU to GPU. (MiB is short for Mebibyte, also called Mega binary byte. The commonly known MB is short for Megabyte and means 1,000,000 Bytes, but 1MB of memory is actually exactly 1,048,576 Bytes. So to avoid confusion when expressing binary values, an "i" is added in between to indicate binary. Mega is 1,0002 and Mebi is 1,0242.)
- GPU-Util 70%: The GPU's current performance, showing how much of 100% performance is being used. In other words, the current usage of that GPU.
- Volatile Uncorr.ECC off: When on, an error count occurs and the number gradually increases from 0. In such cases a system hang occurs, the system stops working and becomes unresponsive, and system operation becomes impossible. So it is usually set to off when working.
- Compute M. off: Shows the Compute-Mode; the Default is 0. 1 means exclusive thread mode, 2 means prohibited mode, and 3 means exclusive process mode.
- MIG M. N/A: A feature for slicing an NVIDIA GPU. MIG M stands for MIG-Mode, and MIG stands for Multi-Instance GPU. When running a CUDA application, it safely partitions the GPU into up to 7 separate GPU instances, providing separate GPUs to multiple users and helping accelerate work. MIG, which supports multiple instances, is supported only on the NVIDIA A100 model, and there are other requirements as well.
- Type(G/C): Both types relate to the GPU.
- G = G for Graphcis; a process that uses the NVIDIA GPU's graphic-mode, used for graphics, professional 3D graphics for video rendering, games, and so on.
- C = C for Compute; a process that uses the NVIDIA GPU's compute-mode with the CUDA library. Used for training and testing deep learning with Tensorflow, Pytorch, and so on.
- C+G = Compute+Graphic combined; a process that uses both compute-mode and graphic-mode.
XML Parsing
Go Version
package nvidia
import "encoding/xml"
type NvidiaSmiLog struct {
XMLName xml.Name `xml:"nvidia_smi_log"`
GPUs []GPU `xml:"gpu"`
}
type GPU struct {
ID string `xml:"id,attr"`
ProductName string `xml:"product_name"`
FanSpeed string `xml:"fan_speed"`
Temperature Temperature `xml:"temperature"`
Utilization Utilization `xml:"utilization"`
FBMemoryUsage FBMemoryUsage `xml:"fb_memory_usage"`
GPUPowerReadings GPUPowerReadings `xml:"gpu_power_readings"`
Processes Processes `xml:"processes"`
}
type Temperature struct {
GPUTemp string `xml:"gpu_temp"`
}
type Utilization struct {
GPUUtil string `xml:"gpu_util"`
}
type FBMemoryUsage struct {
Total string `xml:"total"`
Used string `xml:"used"`
}
type GPUPowerReadings struct {
PowerDraw string `xml:"instant_power_draw"`
CurrentPowerLimit string `xml:"current_power_limit"`
}
type Processes struct {
ProcessInfos []ProcessInfo `xml:"process_info"`
}
type ProcessInfo struct {
PID int `xml:"pid"`
ProcessName string `xml:"process_name"`
UsedMemory string `xml:"used_memory"`
}
func UnmarshalNvidiaSmiLog(data []byte) (*NvidiaSmiLog, error) {
var result NvidiaSmiLog
err := xml.Unmarshal(data, &result)
if err != nil {
return nil, err
} else {
return &result, nil
}
}
Call it roughly like this:
cmd := exec.Command("nvidia-smi", "-q", "-x")
output, err := cmd.Output()
if err != nil {
return
}
smiLog, err := nvidia.UnmarshalNvidiaSmiLog(output)
if err != nil {
log.Printf("Error parsing nvidia-smi XML: %v", err)
return
}
Troubleshooting
Graphics card not recognized after reboot
- (Info) Fixing the "couldn't communicate with the NVIDIA driver" error when running nvidia-smi
- (CUDA ERROR) nvidia-smi has failed because it couldn't communicate with the nvidia driver
- Fixing "nvidia-smi has failed because it couldn't communicate with the nvidia driver", installing the nvidia driver
- Cloud in just seconds | iwinv server
nvidia-smi has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Check installed graphics cards:
Automatically detect installed drivers
Install the recommended driver
| WARNING |
| It does not run properly. |
Insufficient Permissions
$ nvidia-smi -i <GPU-UUID> -c EXCLUSIVE_PROCESS
Unable to set the compute mode for GPU 00000000:5E:00.0: Insufficient Permissions
Terminating early due to previous errors.
Bottom line: use sudo.
Categories
- nvidia
- NvidiaSmi:XmlQueryParser - python code that parses query results output as xml.