NVSentinel

Release v1.30.0

This release adds support for the XID 154 recovery actions of R610 and later drivers, lets the labeler's default GPU class follow a GPU Operator mode switch without manual configuration, adds an option to run metadata-collector without its pod device mapper, and labels platform_connector_health_events_received_total by processing strategy and health. It also fixes the bundled MongoDB network policy for a release that is not named after its namespace, and stops a truncated node condition from staying True after every GPU recovers.

[!NOTE] DCGM 3.x support in GPU Health Monitor is deprecated. It will be removed on 30 November 2026. DCGM 3.x is end-of-life and gets no more patches. After the removal, NVSentinel does not detect faults on nodes that run DCGM 3.x. To find these nodes, run kubectl get nodes -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x. Upgrade DCGM on these nodes to 4.x before you upgrade to the release that removes DCGM 3.x support. For the steps in each global.dcgm.mode, see GPU Health Monitor: DCGM 3.x Support Removal. To give feedback, comment on #2102.

Upgrade Notes

Bundled MongoDB network policy follows the release name (#2186, #2185)

mongodb-networkpolicy selected the bundled Bitnami MongoDB pods by the release namespace, but the pods carry the release name. With the default networkPolicy.enabled: true, the metrics-access policy still isolated the MongoDB pods, so it blocked port 27017, the replica set could not form, and platform-connectors could not reach MongoDB. The policy now selects the pods by release name.

If your release name is different from its namespace and you set networkPolicy.enabled: false, no policy selected the MongoDB pods before. Now mongodb-networkpolicy selects them and allows only port 27017 from the release namespace, so a Prometheus in another namespace can no longer scrape the MongoDB exporter on port 9216. To allow it, add a NetworkPolicy for port 9216, for example through mongodb-store.mongodb.extraDeploy.

Labeler GPU class with --reuse-values (#2108, #1892)

The new default GPU class (see Major New Features) is a change to labeler.expectedDeviceCounts.classes. helm upgrade --reuse-values keeps your previous class list, so GPUCluster mode still needs manual configuration. If you run GPUCluster mode, or switch between modes, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults. If you set your own classes, for example to enable the gpu-dra example class, replace your GPU classes with the new default GPU class, as the labeler configuration docs show. Without sources, a GPU DRA class can report 0 on a node that has only the ResourceSlices of another driver.

Major New Features

R610 and Later Driver Support for XID 154 (#2166, #2174, #2165)

XID 154 reports the recovery action that the GPU driver asks for. R610 and later drivers print PF FLR and OS Reboot where earlier drivers printed GPU Reset Required and Node Reboot Required, and they add Bus Reset and System Reboot. syslog-health-monitor now maps PF FLR to COMPONENT_RESET, and OS Reboot, Bus Reset and System Reboot to RESTART_BM. A bus reset covers more than the single-GPU reset that COMPONENT_RESET runs, so it gets a node reboot. With fault-remediation enabled, NVSentinel resets the GPU or reboots the node for these

Release v1.29.0

This release lets node-drainer select a drain mode by the kind of a pod's controller and adds SELinux support for RKE2 and SUSE Linux Enterprise. It also stops the generic reboot from rebooting a node twice, stops fault-remediation from creating a second maintenance CR after a read error, and fixes node drains that never finished on a bundled MongoDB with fewer than three members. Several behaviours change on upgrade — please read the Upgrade Notes below.

[!NOTE] DCGM 3.x support in GPU Health Monitor is deprecated. It will be removed on 30 November 2026. DCGM 3.x is end-of-life and gets no more patches. After the removal, NVSentinel does not detect faults on nodes that run DCGM 3.x. To find these nodes, run kubectl get nodes -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x. Upgrade DCGM on these nodes to 4.x before you upgrade to the release that removes DCGM 3.x support. For the steps in each global.dcgm.mode, see GPU Health Monitor: DCGM 3.x Support Removal. To give feedback, comment on #2102.

Upgrade Notes

Generic reboot image needs sh and cat (#2110, #2077)

With janitor-provider.csp.provider: generic, every reboot Job now runs its command through sh -c, because the command first checks the node boot ID. The default command was chroot /host reboot, which ran without a shell. The image in janitor-provider.csp.generic.rebootImage must now include sh and cat. The default busybox image has both. If your custom image has no sh, the reboot container does not start and the node does not reboot.

Bundled MongoDB disruption budget (#2112, #2111)

The bundled Bitnami MongoDB set mongodb-store.mongodb.pdb.minAvailable: 2. With fewer than three members, the PodDisruptionBudget allowed no evictions, so a drain of a node that hosted a member never finished. The chart now leaves pdb.minAvailable and pdb.maxUnavailable empty, and the subchart renders maxUnavailable: 1. With one or two members, a drain can now evict a member, and the replica set has no primary until the pod runs again. With more than three members, the budget allows one eviction at a time, where minAvailable: 2 allowed more.

helm upgrade --reuse-values keeps minAvailable: 2. If you run the bundled MongoDB with a member count other than three, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults.

RepeatedXIDErrorOnSameGPU skips XID 45 events (#2064, #2050)

The RepeatedXIDErrorOnSameGPU rule in health-events-analyzer never counted XID 45, but each XID 45 event still started the rule's 24-hour MongoDB query. A job teardown can produce a burst of XID 45 events, and these queries loaded the MongoDB primary and delayed change-stream processing. XID 45 events no longer start the rule.

The fix is in the default value of health-events-analyzer.config. helm upgrade --reuse-values keeps your previous value, so XID 45 events still start the query. If you run health-events-analyzer, upgrade with --reset-then-reuse-values. If you set your own config, copy the two XID 45 exclusions from the RepeatedXIDErrorOnSameGPU rule in the v1.29.0 chart.

Major New Features

Drain Modes by Pod Owner Kind (#2049, #1689)

A node-drainer.podDrainPolicies entry can now select pods by the kind of their controller, with the new `ownerKind

Release v1.28.0

NVSentinel is now graduated.

This release makes metadata-collector and embedded-mode gpu-health-monitor leave a node during GPU Operator driver upgrades and MIG changes, lets MongoDB change-stream consumers recover from transient failures, and adds Kubernetes 1.37 support to preflight gang discovery. It also detects XIDs on MIG GPUs and fixes remediation on Amazon DocumentDB. Several behaviours change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

metadata-collector and embedded gpu-health-monitor follow nvidia.com/gpu.deploy.client (#1988, #1868)

metadata-collector, and gpu-health-monitor in embedded-mode, now have a default required node affinity on the nvidia.com/gpu.deploy.client node label. A pod runs when the label is true or absent. The DaemonSet controller removes the pod for any other value, such as paused-for-driver-upgrade, paused-for-mig-change, or false. While the pod is off the node, embedded-mode gpu-health-monitor does not report GPU health for that node. If a driver upgrade or MIG change fails and leaves the label at a paused value, the pods stay off the node until the label is true again.

The default rule applies only when neither global.affinity nor the component's own affinity is set. global.affinity applies to every NVSentinel component, so if you set it for any reason, these DaemonSets do not get the rule. To keep the rule together with a custom affinity, add its two terms to your affinity, as shown in the metadata-collector and gpu-health-monitor configuration docs.

Remote DCGM probe hangs are store-only by default (#1984, #1978)

In the operator-service and external-hostengine modes, gpu-health-monitor reports a hung DCGM probe as a GpuDcgmConnectivityFailure event with error code DCGM_PROBE_HANG. This event used the global processing strategy (EXECUTE_REMEDIATION by default), so the default rules cordoned the node. It now follows gpu-health-monitor.dcgm.probeStoreOnly, which defaults to true, so a remote probe hang is stored and NVSentinel does not act on it. Other DCGM connectivity errors still use the global strategy. To restore remediation for remote probe hangs, set probeStoreOnly: false. That setting also enables remediation for the embedded-mode GpuDcgmUnresponsive event, which recommends RESTART_BM.

MongoDB metrics in a namespace other than nvsentinel (#2053, #2052)

The bundled Bitnami MongoDB metrics exporter connected to mongodb-headless.nvsentinel.svc in every namespace, so MongoDB metrics were missing when NVSentinel ran in any other namespace. The fix changes the default value of mongodb-store.mongodb.metrics.args. helm upgrade --reuse-values keeps your previous value, so the exporter still connects to the wrong namespace. If you run the bundled MongoDB in a namespace other than nvsentinel, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults.

Major New Features

GPU Operator Driver Upgrades and MIG Changes (#1988, #1868)

GPU Operator 26.7 and later uses the nvidia.com/gpu.deploy.client node label to move GPU clients off a node before it unloads the driver or changes the MIG layout. metadata-collector and embedded-mode gpu-health-monitor keep the GPUs open, but they ignored this label, so during a driver upgrade they could keep the old driver in use. Both DaemonSets now leave the node while the label holds any value other than true, and return when it is true again. Older

Release v1.27.0

This release adds GPU Operator GPUCluster (DRA) mode support to janitor GPU reset, lifecycle-manager, node-drainer, and preflight. It also adds operator recovery for health-events-analyzer conditions, poll-stall detection for the NIC health monitor, and a percentage form of the standing cordon bound. Several defaults and behaviours change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

GpuThermalMarginWatch requires three consecutive polls (#1982, #1980)

GpuThermalMarginWatch now fails a GPU only when the thermal margin stays below the slowdown threshold for gpu-health-monitor.dcgmFieldsMonitoring.gpuTempLimitMinConsecutivePolls consecutive polls. The default is 3, which is 45 seconds at the default 15-second poll interval. A sample at or above the threshold resets the count. The check reads one sample per poll, and on GB200 under load the margin moves by tens of degrees within seconds. A single sample past the threshold is therefore usually a transient that the GPU's own hardware slowdown already handled. To restore the previous single-sample behaviour, set the value to 1.

NVLink BER error codes are suppressed by default (#1994, #1944)

DCGM 4.7.0 removed the NVLink BER health and diagnostic checks because they failed healthy GPUs. Older DCGM hostengines still report DCGM_FR_NVLINK_EFFECTIVE_BER_THRESHOLD and DCGM_FR_NVLINK_SYMBOL_BER_THRESHOLD. Both map to CONTACT_SUPPORT, so the default quarantine rules cordon the node. Both codes are now in the default gpu-health-monitor.dcgmHealthCheck.suppressedErrorCodes list. On DCGM 4.7.0 and later, this has no effect.

The new default does not reach every installation. helm upgrade --reuse-values keeps the list from your previous release, and a custom suppressedErrorCodes list replaces the default list. In both cases, add the two codes to your list.

Post-remediation validation (#1962, #1960)

These changes apply only when post-remediation validation is enabled (fault-quarantine.validation.enabled).

Quarantine taints stay on the node during validation. Fault-quarantine kept the cordon when it handed a node to validation, but it removed its quarantine taints, so a node with taint-only rule-sets accepted workloads while validation ran. It now keeps the taints too. After validation passes, lifecycle-manager lifts a taint only if lifecycle-manager.config.schedulingGate.taints lists it with remove: true. Any other taint stays on the node, and test pods do not tolerate it unless it is a NoExecute taint. Add every taint that your fault-quarantine rule-sets apply to that list, as described in Validation configuration. With the default rule-sets, this affects the certification-failure taint nvsentinel.dgxc.nvidia.com/nvcre-cert-failed. lifecycle-manager matches taints on key, value, and effect, so with remove: true it also lifts a matching taint that was on the node before the quarantine.

Partial drain must be configured for both components. Fault-quarantine now reads its own partial-drain setting, fault-quarantine.validation.partialDrainEnabled, which defaults to false. If you set node-drainer.partialDrainEnabled: true, fault-quarantine can count a partial drain as a full drain and start validation before the node is fully drained. Set global.partialDrainEnabled: true instead. It sets both components.

Janitor GPU reset with global.gpuDraEnabled (#1989, #1859)

GPUCluster (DRA)