Nvidia-smi-exporter

lastmod 2025-12-03created 2025-12-029 edits

Nvidia-smi Prometheus exporter with respecting of GPU-UUID

Collected metric categories

System and basic GPU information

nvidia_smi_attached_gpus - number of attached GPUs
nvidia_smi_display_mode - display mode state (0=Disabled, 1=Enabled)
nvidia_smi_display_active - display active state (0=Not Active, 1=Active)
nvidia_smi_persistence_mode - persistence mode state (0=Disabled, 1=Enabled)
nvidia_smi_minor_number - GPU minor device number
nvidia_smi_multigpu_board - whether it is a multi-GPU board (0=No, 1=Yes)

Accounting information

nvidia_smi_accounting_mode - accounting mode state (0=Disabled, 1=Enabled)
nvidia_smi_accounting_mode_buffer_size - accounting mode buffer size

InfoROM information

nvidia_smi_inforom_version_oem_object - OEM object InfoROM version

PCI information

nvidia_smi_pci_pci_sub_system_id - PCI subsystem ID
nvidia_smi_pci_pci_gpu_link_info_pcie_gen_max_link_gen - maximum PCIe generation
nvidia_smi_pci_pci_gpu_link_info_pcie_gen_current_link_gen - current PCIe generation
nvidia_smi_pci_pci_gpu_link_info_link_widths_max_link_width - maximum PCIe link width (1x, 2x, 4x, 8x, 16x, 32x)
nvidia_smi_pci_pci_gpu_link_info_link_widths_current_link_width - current PCIe link width
nvidia_smi_pci_replay_counter - PCI replay counter
nvidia_smi_pci_replay_rollover_counter - PCI replay rollover counter
nvidia_smi_pci_tx_util_bytes_per_second - PCI TX bandwidth usage (bytes/s)
nvidia_smi_pci_rx_util_bytes_per_second - PCI RX bandwidth usage (bytes/s)

Fan and performance state

nvidia_smi_fan_speed_ratio - fan speed ratio (0.0~1.0)
nvidia_smi_performance_state - performance state (P0P15, converted to 015)

Clock throttling information

nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_gpu_idle - throttling due to GPU idle
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_applications_clocks_setting - throttling due to application clock settings
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_sw_power_cap - throttling due to software power cap
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_hw_slowdown - hardware slowdown
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_hw_thermal_slowdown - hardware thermal slowdown
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_hw_power_brake_slowdown - hardware power brake slowdown
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_sync_boost - sync boost
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_sw_thermal_slowdown - software thermal slowdown
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_display_clocks_setting - display clock setting

Memory usage

nvidia_smi_fb_memory_usage_total_bytes - total framebuffer memory (bytes)
nvidia_smi_fb_memory_usage_used_bytes - framebuffer memory in use (bytes)
nvidia_smi_fb_memory_usage_free_bytes - available framebuffer memory (bytes)
nvidia_smi_bar1_memory_usage_total_bytes - total BAR1 memory (bytes)
nvidia_smi_bar1_memory_usage_used_bytes - BAR1 memory in use (bytes)
nvidia_smi_bar1_memory_usage_free_bytes - available BAR1 memory (bytes)

Compute mode

nvidia_smi_compute_mode - compute mode (0=Default, 1=Exclusive_Thread, 2=Prohibited, 3=Exclusive_Process)

Utilization

nvidia_smi_utilization_gpu_util_ratio - GPU core utilization ratio (0.0~1.0)
nvidia_smi_utilization_memory_util_ratio - memory utilization ratio (0.0~1.0)
nvidia_smi_utilization_encoder_util_ratio - encoder utilization ratio (0.0~1.0)
nvidia_smi_utilization_decoder_util_ratio - decoder utilization ratio (0.0~1.0)

Encoder statistics

nvidia_smi_encoder_stats_session_count - number of active encoder sessions
nvidia_smi_encoder_stats_average_fps - average encoder FPS
nvidia_smi_encoder_stats_average_latency - average encoder latency

FBC (Frame Buffer Capture) statistics

nvidia_smi_fbc_stats_session_count - number of active FBC sessions
nvidia_smi_fbc_stats_average_fps - average FBC FPS
nvidia_smi_fbc_stats_average_latency - average FBC latency

Temperature

nvidia_smi_temperature_gpu_temp_celsius - current GPU temperature (Celsius)
nvidia_smi_temperature_gpu_temp_max_threshold_celsius - GPU maximum threshold temperature (Celsius)
nvidia_smi_temperature_gpu_temp_slow_threshold_celsius - GPU slowdown threshold temperature (Celsius)
nvidia_smi_temperature_gpu_target_temperature_celsius - GPU target temperature (Celsius)
nvidia_smi_supported_gpu_target_temp_gpu_target_temp_min_celsius - minimum supported GPU target temperature (Celsius)
nvidia_smi_supported_gpu_target_temp_gpu_target_temp_max_celsius - maximum supported GPU target temperature (Celsius)

Power information

nvidia_smi_power_readings_power_state - power state (P0~P15)
nvidia_smi_power_readings_power_management - whether power management is enabled (0=Disabled, 1=Enabled)
nvidia_smi_power_readings_power_draw_watts - current power draw (watts)
nvidia_smi_power_readings_power_limit_watts - current power limit (watts)
nvidia_smi_power_readings_default_power_limit_watts - default power limit (watts)
nvidia_smi_power_readings_enforced_power_limit_watts - enforced power limit (watts)
nvidia_smi_power_readings_min_power_limit_watts - minimum power limit (watts)
nvidia_smi_power_readings_max_power_limit_watts - maximum power limit (watts)

Clock speeds

nvidia_smi_clocks_graphics_clock_hz - current graphics clock (Hz)
nvidia_smi_clocks_sm_clock_hz - current SM (streaming multiprocessor) clock (Hz)
nvidia_smi_clocks_mem_clock_hz - current memory clock (Hz)
nvidia_smi_clocks_video_clock_hz - current video clock (Hz)
nvidia_smi_max_clocks_graphics_clock_hz - maximum graphics clock (Hz)
nvidia_smi_max_clocks_sm_clock_hz - maximum SM clock (Hz)
nvidia_smi_max_clocks_mem_clock_hz - maximum memory clock (Hz)
nvidia_smi_max_clocks_video_clock_hz - maximum video clock (Hz)

Memory details

nvidia_smi_fb_memory_usage_total_bytes (Frame Buffer Memory)

The total size of the GPU's actual physical VRAM
The "Total" memory shown in nvidia-smi is exactly this value
e.g. 24GB for an RTX 3090

nvidia_smi_bar1_memory_usage_total_bytes (BAR1 Memory)

Size of the PCIe BAR1 address space
A region mapped so that the CPU can access GPU memory directly
A portion of FB Memory mapped into the CPU address space
Usually much smaller than FB Memory (e.g. 256MB to a few GB)

Running from source code

The basic prerequisite is that the nvidia-smi command works.

First, if you do not have a ruby environment:

sudo apt install rbenv
echo 'eval "$(rbenv init -)"' >> ~/.bashrc

Written for Ubuntu 22.04:

## 코드 복사
git clone https://github.com/a0s/nvidia-smi-exporter
cd nvidia-smi-exporter

## 루비 환경 설정
rbenv install 2.7.1
rbenv local 2.7.1

## 버전 2.7.1 확인
ruby -v

## Bundler 설치
gem install bundler -v 2.4.22

## 종속성 설치
bundle install

## 실행
ruby ./app/application.rb

You should see log output like the following.

== Sinatra (v2.2.0) has taken the stage on 9454 for development with backup from Puma
Puma starting in single mode...
* Puma version: 5.6.4 (ruby 2.7.1-p83) ("Birdie's Version")
*  Min threads: 0
*  Max threads: 5
*  Environment: development
*          PID: 301532
* Listening on http://0.0.0.0:9454
Use Ctrl-C to stop

Metrics example

The result of curl localhost:9454/metrics is as follows:

nvidia_smi_attached_gpus 2
nvidia_smi_display_mode{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1
nvidia_smi_display_active{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_persistence_mode{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1
nvidia_smi_accounting_mode{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_accounting_mode_buffer_size{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 4000
nvidia_smi_minor_number{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_multigpu_board{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_inforom_version_oem_object{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1.1
nvidia_smi_pci_pci_sub_system_id{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 37021458
nvidia_smi_pci_pci_gpu_link_info_pcie_gen_max_link_gen{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 3
nvidia_smi_pci_pci_gpu_link_info_pcie_gen_current_link_gen{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 3
nvidia_smi_pci_pci_gpu_link_info_link_widths_max_link_width{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 16
nvidia_smi_pci_pci_gpu_link_info_link_widths_current_link_width{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1
nvidia_smi_pci_replay_counter{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 65535
nvidia_smi_pci_replay_rollover_counter{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_pci_tx_util_bytes_per_second{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 11264000
nvidia_smi_pci_rx_util_bytes_per_second{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 31744000
nvidia_smi_fan_speed_ratio{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0.53
nvidia_smi_performance_state{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 2
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_gpu_idle{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_applications_clocks_setting{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_sw_power_cap{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_hw_slowdown{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_hw_thermal_slowdown{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_hw_power_brake_slowdown{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_sync_boost{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_sw_thermal_slowdown{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_clocks_throttle_reasons_clocks_throttle_reason_display_clocks_setting{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_fb_memory_usage_total_bytes{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 8510242816
nvidia_smi_fb_memory_usage_used_bytes{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 3040870400
nvidia_smi_fb_memory_usage_free_bytes{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 5469372416
nvidia_smi_bar1_memory_usage_total_bytes{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 268435456
nvidia_smi_bar1_memory_usage_used_bytes{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 5242880
nvidia_smi_bar1_memory_usage_free_bytes{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 263192576
nvidia_smi_compute_mode{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_utilization_gpu_util_ratio{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1.0
nvidia_smi_utilization_memory_util_ratio{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1.0
nvidia_smi_utilization_encoder_util_ratio{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0.0
nvidia_smi_utilization_decoder_util_ratio{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0.0
nvidia_smi_encoder_stats_session_count{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_encoder_stats_average_fps{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_encoder_stats_average_latency{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_fbc_stats_session_count{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_fbc_stats_average_fps{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_fbc_stats_average_latency{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 0
nvidia_smi_temperature_gpu_temp_celsius{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 67.0
nvidia_smi_temperature_gpu_temp_max_threshold_celsius{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 96.0
nvidia_smi_temperature_gpu_temp_slow_threshold_celsius{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 93.0
nvidia_smi_temperature_gpu_target_temperature_celsius{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 83.0
nvidia_smi_supported_gpu_target_temp_gpu_target_temp_min_celsius{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 60.0
nvidia_smi_supported_gpu_target_temp_gpu_target_temp_max_celsius{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 92.0
nvidia_smi_power_readings_power_state{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 2
nvidia_smi_power_readings_power_management{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1
nvidia_smi_power_readings_power_draw_watts{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 211.86
nvidia_smi_power_readings_power_limit_watts{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 216.0
nvidia_smi_power_readings_default_power_limit_watts{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 200.0
nvidia_smi_power_readings_enforced_power_limit_watts{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 216.0
nvidia_smi_power_readings_min_power_limit_watts{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 100.0
nvidia_smi_power_readings_max_power_limit_watts{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 216.0
nvidia_smi_clocks_graphics_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1784000000.0
nvidia_smi_clocks_sm_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1784000000.0
nvidia_smi_clocks_mem_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 5514000000.0
nvidia_smi_clocks_video_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1594000000.0
nvidia_smi_max_clocks_graphics_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1999000000.0
nvidia_smi_max_clocks_sm_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1999000000.0
nvidia_smi_max_clocks_mem_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 5005000000.0
nvidia_smi_max_clocks_video_clock_hz{uuid="683c6e09-3969-31a5-b7ca-cc88252b7fce"} 1708000000.0

Favorite site

Github - a0s/nvidia-smi-exporter

author Zer0 sitemap