In this article
CDI device not found calls for checking the qualified name and effective device specification. Compare the requested name with nvidia-ctk cdi list first. Seeing a GPU through host nvidia-smi does not establish that the container engine has an up-to-date CDI definition.
CDI device not found calls for checking the qualified name and effective device specification. Compare the requested name with nvidia-ctk cdi list first. Seeing a GPU through host nvidia-smi does not establish that the container engine has an up-to-date CDI definition.
Identify the injection mode
Record NVIDIA Container Toolkit, engine version and launch command. CDI describes devices declaratively, including nodes, mounts and hooks. A name such as nvidia.com/gpu=0 belongs to that specification rather than being an ordinary /dev path. Check engine support and discovery directories. Docker documents when CDI became enabled by default, but an upgraded host can retain different effective settings.
Match the list and specification
Run nvidia-ctk cdi list and locate the active YAML or JSON definition. Compare its devices with host inventory, readability and the engine’s discovery path. Recent Toolkit versions support automatic generation through nvidia-cdi-refresh. That mechanism has documented limitations, including particular MIG changes. Where applicable, preserve service/path unit state and generation logs. A successful generator does not establish that another daemon reads the same directory.
Examine a stale definition
In an example, device configuration changes while a container requests the previous qualified name. Host enumeration reflects the new state but CDI data remains old. Correctly refreshing the specification makes selection verifiable again. Do not assume identifiers retain their former meaning after a change. Compare UUID or MIG identity for the new definition rather than relying solely on a convenient ordinal in the command.
| Check | Evidence required |
|---|---|
| Requested name | Exact —device argument |
| Discovery | Directory read by the engine |
| Inventory | CDI list and host identities |
| Refresh | Successful current generation |
Exclude duplicate injection
NVIDIA documents potential conflict between CDI and the legacy runtime hook. Inspect OCI hooks and NVIDIA_VISIBLE_DEVICES usage rather than accidentally combining mechanisms. Do not delete system files blindly; establish which workloads rely on the present mode. Preserve configuration and rollback before a transition. One successful container does not mean every daemon or runtime on the host has adopted the new method.
Verify a fresh launch
Create a separate test container selecting one CDI device name and compare its visible UUID with the expected device. Repeat after the intended refresh or test reboot procedure. Retain versions, specification path, selected name and observed identity. Avoid expanding permissions merely to make the demonstration work. Correct CDI selection confirms device injection; computation and application compatibility remain independent next checks. This distinction keeps an inventory repair from being mistaken for validation of the full inference environment.
Keep the qualified name beside actual identity. Recheck that relationship after inventory changes so an old launch argument does not silently identify a different device.
Related operational reading: gpu containers vs vms; server preparation.