In this article
Failed to initialize NVML: Unknown Error after systemctl daemon-reload can match a documented legacy NVIDIA runtime-hook issue involving cgroup updates. Check mode and versions: the same error has other causes, so matching the message is insufficient for diagnosis.
Failed to initialize NVML: Unknown Error after systemctl daemon-reload can match a documented legacy NVIDIA runtime-hook issue involving cgroup updates. Check mode and versions: the same error has other causes, so matching the message is insufficient for diagnosis.
Preserve the event sequence
Record the last successful access, daemon-reload or resource update, and first failure. Compare host nvidia-smi with access inside the affected container. If the host also cannot access the device, investigation extends beyond container injection. Do not trigger daemon-reload in production merely to test a guess. Start with existing maintenance records and arrange reproduction on a test node with an acceptable impact boundary.
Identify device injection
NVIDIA describes legacy hooks modifying device and cgroup access outside the low-level runtime’s full awareness. A later container update can remove GPU access. On certain systemd cgroup configurations, daemon-reload is sufficient to trigger such an update. runc and driver versions affect particular variants. Preserve runtime mode, cgroup driver, versions and device selection. CDI uses a different injection model, so legacy-mode conclusions must not be transferred automatically.
Separate availability layers
In an example, host enumeration remains normal while an existing container loses NVML access after maintenance. A newly created test container with the same assignment sees the GPU again. This supports a container-access hypothesis when mode and events also match. Restarting only the application inside the affected container may leave its runtime configuration unchanged. Documentation describes container recreation as recovery for this particular case.
| Observation | Meaning for the hypothesis |
|---|---|
| Host sees the device | Investigate beyond host access |
| Existing container cannot | Inspect runtime device access |
| New container can | Compare created configurations |
| Cgroup update occurred | Check the documented mechanism |
Choose a compatible mitigation
Documentation lists mitigations including CDI and explicit device-node handling. Do not change the entire cluster’s cgroup driver based on one symptom; it interacts with kubelet, runtime and node policy. Select a supported approach for the environment, preserve configuration and assess neighboring workloads. Moving to CDI requires discovery and specification checks, not merely substituting one launch flag.
Repeat the maintenance test
On a test node, repeat the intended update and confirm access before and after it. Record versions, injection mode, UUID and a small compute check separately from nvidia-smi. The artifact should identify the reproduced scenario and selected mitigation. If failure occurs without a cgroup update or only in one application, continue investigating libraries, permissions and other causes. A narrowly documented defect should not become a universal explanation for every NVML error.
Related operational reading: gpu maintenance change register; vllm install compatibility.