需使用与该 exporter 版本配对的 DCGM 版本,版本不匹配可能无法正常工作;README 未给出具体运行命令,安装与配置需参考 NVIDIA 官方文档;容器镜像使用受 NVIDIA Software License Agreement 约束,与源码的 Apache-2.0 不同。
docker run --gpus all -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:latestREADME 未说明具体显存或 GPU 型号要求,仅说明依赖 NVIDIA Data Center GPU Manager (DCGM) 采集 GPU 遥测数据。
把 NVIDIA GPU 的 DCGM 遥测数据转成 Prometheus 格式,方便监控 GPU 集群
把 NVIDIA GPU 的硬件遥测数据导出成 Prometheus 指标,方便监控显卡温度、功耗、显存和利用率等运行状态。
核心能力
- 基于 DCGM 采集 NVIDIA GPU 遥测指标
- 以 Prometheus 文本格式暴露指标
- 提供容器镜像、软件包和 Helm chart 部署方式
- 附带 Grafana 仪表盘 JSON 可直接导入
- 默认指标定义在 etc/default-counters.csv 中
容易踩的坑
- 4.9.0最新正式版2026-11-06 · 2026-11-06 18:50 UTC
Tested component versions: DCGM 4.7.0, go-dcgm v1.4712.2, go-nvml v0.13.4-1, Go 1.27.3, and NVIDIA distroless CC v4.2.0.
DCGM Exporter 4.9.0 is tested with DCGM 4.7.0 and 4.7.1 and ships with DCGM 4.7.0. In DCGM 4.7.1, some requests between libdcgm and the host engine fail with an API version mismatch when one side is 4.7.1 and the other is an earlier version. If you use DCGM 4.7.1, run a 4.7.1 host engine with a 4.7.1 libdcgm.
4.9.0 renames default metrics and labels, changes some units, and changes the configuration and Helm values formats. Read the upgrade checklist in the DCGM Exporter 4.9.0 release notes before upgrading from 4.8.4.
Metrics
- Update to DCGM 4.7.0 and go-dcgm v1.4712.2.
- Reject malformed DCGM responses that used to crash the exporter or produce wrong values.
- Rename the 23 default and profiling-preset fields that DCGM 4.7 deprecates to their new DCGM 4.7 names. Eleven of them also change unit: clocks from MHz to hertz, energy from millijoules to joules, utilization from a percentage to a 0 to 1 ratio, GPU memory from MiB to bytes, and NVLink throughput from MB/s to bytes per second.
- Rename the driver-version label on every metric from
DCGM_FI_DRIVER_VERSIONtoDCGM_FI_SYSTEM_DRIVER_VERSION. Helm installs did not have this label in 4.8.4 and now get it. - Collect
DCGM_FI_DEV_FB_CAPACITY_BYTES(total GPU memory in bytes) by default. - Deprecate
dcp-metrics-included.csvand1.x-compatibility-metrics.csv. The exporter logs a warning at startup when either file is used. Usedefault-counters.csvinstead. - Remove commented-out rows that use deprecated or old field names from the shipped CSV files. The remaining commented-out rows list optional fields under their DCGM 4.7 names.
- Add optional fields in hertz, bytes, seconds, and 0 to 1 ratios for measurements that the default metric list does not include.
- For Kubernetes pods, add up per-process GPU utilization and GPU memory use under the new
DCGM_FI_DEV_GPU_UTIL_RATIOandDCGM_FI_DEV_FB_USED_BYTESnames as well as the old names. A pod's utilization ratio can be above 1.0. - Update the bundled Grafana dashboard to the new metric names and units.
- Add optional
DCGM_EXP_FIELD_POLICY_*gauges that report each DCGM policy's state, thresholds, current value, violation count, and last trigger time. - Report XID count and total metrics for each selected MIG GPU instance, its compute instances, and its parent GPU (#751 by @sylvesterkaczmarek; fixes #750, reported by @erikkressner).
- When the exporter monitors MIG instances (the default device selection), report GPU-wide fields such as power, temperature, clocks, and PCIe once per physical GPU instead of repeating them for each MIG instance. These metrics have no MIG instance labels and carry pod labels only when one pod owns the whole GPU, so adding them up across instances no longer counts a GPU more than once. The clock-event count and total metrics work the same way. Addresses #561, reported by @taejune, and #353, reported by @frittentheke.
- Set
NVIDIA_MIG_MONITOR_DEVICES=allin container images so DCGM can collect profiling fields on MIG-enabled GPUs and their instances. This fills in the default PCIe TX/RX and DRAM utilization metrics, which the 4.8.4 image left blank on MIG-enabled GPUs. - Reject raw fields that cannot be exported safely, such as bitmasks, supported-clock lists, and status fields that can read as a misleading zero after a DCGM error. Reject help text that is not valid UTF-8.
- Keep the physical link numbers on GPU NVLink metrics, including gaps where links are inactive (fixes #753, reported by @v0ropaev).
- Apply `--
- 4.8.42026-09-18 · 2026-09-18 20:35 UTC
查看更新说明
Tested component versions: DCGM 4.6.1, go-dcgm v1.4611.2, go-nvml v0.13.4-0, Go 1.27.1, and NVIDIA distroless CC v4.1.4.
Runtime and performance
- Release DCGM Exporter 4.8.4 with DCGM 4.6.1 and go-dcgm v1.4611.2.
- Add configurable DCGM field-watch retention limits through CLI, environment, and YAML configuration.
- Add the version 2 YAML schema with plural
collectionswhile preserving version 1 compatibility. - Add configurable limits and request coalescing for concurrent
/metricsscrapes. - Reduce allocations during metric gathering, transformation, and rendering.
- Support metric configurations containing more than 128 DCGM fields by batching reads and splitting oversized field groups.
- Add optional GPU bind/unbind detection that resets DCGM and NVML, then rebuilds metrics for the new GPU topology (#706, reported by @varunrsekar; #716 by @dharmjit).
- Honor
LD_LIBRARY_PATHwhen locating DCGM libraries.
Metrics and Kubernetes
- Add opt-in Go runtime, process, and HTTP handler metrics through
--enable-exporter-metrics,DCGM_EXPORTER_ENABLE_EXPORTER_METRICS, or YAML configuration. - Add opt-in GPU fabric health fields:
DCGM_FI_DEV_FABRIC_HEALTH_SUMMARYandDCGM_FI_DEV_FABRIC_HEALTH_MASK. - Add opt-in configured and enforced power-limit fields:
DCGM_FI_DEV_POWER_MGMT_LIMITandDCGM_FI_DEV_ENFORCED_POWER_LIMIT(#719 by @kumaraditya18027). - Refresh the optional NVLink error-counter definitions.
- Remove retired PCIe throughput fields from the supplied metric configurations. Custom configurations that use them fail and point to
DCGM_FI_PROF_PCIE_TX_BYTESandDCGM_FI_PROF_PCIE_RX_BYTES. - Collect compute-instance-scoped DCGM fields from MIG compute instances and add the
GPU_CI_IDlabel while preserving whole-GPU collection. - Attribute DRA MIG metrics to pods when the MIG device is visible to DCGM at startup, and refresh allocation metadata when ResourceSlices change (#714, reported by @guptaNswati).
- Require an exporter restart after live Dynamic MIG creation or removal; changing the MIG layout while the exporter is running is not supported.
- Add the optional
pod_uidlabel to DRA-attributed metrics (#738 by @git-jxj; fixes #718). - Reuse the kubelet pod-resources gRPC connection to prevent memory growth (#705 by @AshwinRamaniPsg).
- Prefer NVML MIG profile names and make MIG profile lookup available outside Kubernetes (#713, reported by @rahulait).
- Deduplicate repeated job IDs within each HPC mapping file while preserving mappings across files.
- Warn when HPC job mapping entries do not match discovered MIG instances.
Helm, containers, and packages
- Include all supported metric preset CSV files in binary packages.
- Include third-party notices in archives, packages, and container images.
- Include the dynamic linker cache in the distroless image so DCGM libraries can be found at startup (#724 by @arielrahamim; fixes #612).
- Add Helm values for DaemonSet annotations (#723 by @ranyhb).
- Add Helm values for readiness and liveness probe timing and failure thresholds (#679 by @ChenZiHong-Gavin).
- Render debug dump compression explicitly, including when it is disabled.
- Document opt-in scheduling on nodes with the
nvidia.com/gpu:NoScheduletaint (#733 by @KR-Ravindra).
- 4.6.0-4.8.32026-07-15 · 2026-07-15 17:31 UTC
查看更新说明
Changelist
Runtime and configuration
- Release DCGM Exporter 4.8.3 with DCGM 4.6.0 and go-dcgm v1.4601.1 (@nccurry).
- Improve go-dcgm policy, resource, field, and version API handling.
- Add YAML configuration-file support.
- Expand CLI and environment configuration.
- Add configurable web read/write timeouts.
- Add per-watch polling intervals.
- Add combined device selectors.
- Add VSOCK remote hostengine connections.
- Use the local hostname for loopback remote hostengines (#697 by @skartikey).
- Improve hot reload and last-known-good configuration handling.
- Improve profiling and watcher recovery.
- Improve startup, skipped-field, MIG, and remote-hostengine diagnostics.
- Strengthen opt-in pprof and socket validation.
Metrics and Kubernetes
- Rename the
Hostnamelabel tohostname(#655 by @daphne2337). - Add generic container-runtime label collection.
- Support pod-label collection on MIG devices (#660 by @Raid57).
- Allocate independent label maps for each entity (#663 by @rocker-zhang).
- Add DRA ResourceSlice v1 support (#654 by @adityasingh0510).
- Expand DRA v1 and MIG allocation handling (#664 by @guptaNswati).
- Bound pod-resources responses to prevent resource-exhaustion errors (#510 by @halcyon-r).
- Improve Kubernetes metadata diagnostics.
- Add cumulative XID error totals.
- Add cumulative clock-event totals.
- Add GPU-health severity and category metadata.
- Add Grace CPU serial labels.
- Improve metric help text and metadata consistency (@rvatkar, @dshaiknvidia).
- Correct NVLink bandwidth metrics from counters to gauges (#658 by @allenz92).
- Harden metric float parsing (#489 by @tongyangyeyue).
- Improve field-status handling, metric exposition, and label sanitization.
- Propagate HTTP request context through metric rendering (#687 by @randomizedcoder).
Helm, containers, and packages
- Add Helm support for configuration files.
- Add Helm image-digest configuration.
- Update ServiceMonitor timing defaults.
- Align RBAC with Kubernetes metadata and DRA features.
- Make service-account token mounting configurable (#665 by @75asu).
- Send package-managed systemd logs to journald (#671).
- Correct container image provenance metadata (#678 by @shravyavorugallu).
- Improve DCGM, NVPerf, and NVSDM library packaging (#704 by @daveoy, #703).
- Complete the Go 1.26.4 update (#641, #667).
- Remove the direct dependency on
github.com/pkg/errors(#699 by @skartikey). - Centralize release versions and consistency validation.
Testing and documentation
- Add a reusable E2E CLI and scenario framework.
- Expand static, host, container, and Kubernetes test coverage.
- Add repository-local agent-assisted development guidance.
- Add the
llms.txtmetrics and deployment authoring contract. - Add Codex and Cursor project guidance.
- 4.5.3-4.8.22026-05-07 · 2026-05-07 00:28 UTC
查看更新说明
- Update to DCGM 4.5.3 and DCGM Exporter 4.8.2.
- Improve GPU health metrics, including reporting GPU-wide health incidents such as fallen-off-bus XIDs.
- Make
/debug/pprofprofiling endpoints opt-in via--enable-pprof/DCGM_EXPORTER_ENABLE_PPROF. - Add PodMapper informer caching for Kubernetes pod mapping (#626) (@jaeeyoungkim).
- Add per-process GPU metrics for time-sharing and MIG (#594) (@krystiancastai).
- Make Helm
priorityClassNameconfigurable with explicit defaults (#444) (@runzhliu). - Add MIG device support for HPC job labels (#602) (@jay-mckay).
- Update go-dcgm field metadata handling, deprecated field alias resolution, health constants, policy registration handling, and version info APIs.
- Document IPv6 address formats for remote hostengine and metrics listen addresses.
- Refresh dependencies, container base images, Docker image references, Helm chart values, Kubernetes manifests, and tests for this release.
- 4.5.2-4.8.12026-02-09 · 2026-02-09 15:43 UTC
查看更新说明
- Update to DCGM 4.5.2, latest Go 1.24, and base containers
- Fix distroless symlink issue
- Fix for parsing blank XIDs
- Fix for nvlink entities starting at offset 1
用户点评
登录用户的真实使用记录,可按版本和硬件筛选还没有人点评。第一个说出真实体验的人最有价值。
DCGM Exporter
NVIDIA hardware telemetry for Prometheus.
Get started · Command reference · Deploy with Helm · Contribute
Overview
DCGM Exporter exposes NVIDIA GPU telemetry from NVIDIA Data Center GPU Manager (DCGM) in the Prometheus text format. This repository contains the exporter source, container and package builds, Helm chart, tests, and a Grafana dashboard.
Documentation
Use the official NVIDIA documentation for user guidance:
- Install DCGM Exporter
- Configure Prometheus for DCGM Exporter
dcgm-exportercommand reference- DCGM Exporter Metrics
- DCGM Exporter Release Notes
The installation guide covers deployment, configuration, verification, administration, and troubleshooting.
Version compatibility
Run DCGM Exporter with the DCGM version paired with that exporter release. See Version Compatibility for supported package and remote-connection combinations.
Repository resources
- Helm chart configuration and troubleshooting
- Contribution requirements and development workflow
- Kubernetes integration and end-to-end tests
- Container tests and image examples
- CLI package maintainer note
- Default metric definitions
Dashboards
- Checked-in Grafana dashboard:
grafana/dcgm-exporter-dashboard.json
Contributing and support
Read CONTRIBUTING.md before submitting a change. For community support, open an issue. NVIDIA Enterprise customers can access knowledge base articles and support cases, or submit a ticket.
Report potential security vulnerabilities through the NVIDIA Product Security process, not through a public issue. See SECURITY.md for reporting details.
License
DCGM Exporter source is licensed under the Apache License, Version 2.0. Use of the NGC container image is also subject to the NVIDIA Software License Agreement.