In this article
NVIDIA_VISIBLE_DEVICES controls which GPUs the runtime provides to a container. CUDA_VISIBLE_DEVICES filters and reorders devices for a CUDA application within its available environment. A process device number can therefore differ from the host index without indicating incorrect assignment.
NVIDIA_VISIBLE_DEVICES controls which GPUs the runtime provides to a container. CUDA_VISIBLE_DEVICES filters and reorders devices for a CUDA application within its available environment. A process device number can therefore differ from the host index without indicating incorrect assignment.
Begin with stable identity
On the host, record GPU UUIDs and their corresponding indexes. Obtain the container device list and CUDA-process enumeration separately. An index is a position in a local list; a UUID helps correlate observations across scopes. Do not assume device zero always identifies the same physical GPU across nodes or changes. For MIG, also record the instance identity and enumeration constraints of the installed CUDA version.
Inspect the runtime boundary
Docker may use —gpus, NVIDIA runtime environment settings or CDI, depending on mode and version. NVIDIA_VISIBLE_DEVICES controls injected devices and has special values including all, none and void with different behavior. Do not apply those meanings automatically to CUDA_VISIBLE_DEVICES. Preserve effective launch settings, runtime mode and device list. A process variable cannot create a GPU the runtime never provided or replace permission to access it.
Understand local numbering
In an illustrative CUDA environment, CUDA_VISIBLE_DEVICES=2,1 selects those available devices and changes their order: local ordinal 0 represents the first selection. The example assumes those identifiers exist in the relevant environment. For a container, reconstruct its own list before applying the process filter. Do not insert host indexes without mapping when only a subset was injected. UUIDs make the relationship more explicit.
| Scope | Evidence to retain |
|---|---|
| Host | Index, UUID and device list |
| Container | Devices supplied by the runtime |
| CUDA process | Filter and local ordinals |
| Scheduler | The intended assignment |
Inspect process launch settings
An interactive shell can have different variables from a service or ENTRYPOINT. Capture approved keys rather than dumping secrets with the full environment. Check whether a wrapper overrides CUDA_VISIBLE_DEVICES after container startup. Map each worker process independently. Visibility is not a complete memory-isolation mechanism or a performance quota; it selects devices, while sharing requires separate policy and verification.
Confirm assignment with computation
Correlate the expected UUID with a small agreed CUDA workload. nvidia-smi and CUDA enumeration answer different questions, so retain both observations. The artifact should connect host identity, container configuration and process mapping. Recheck after deployment changes, especially when device inventory changes. Correct numbering resolves assignment ambiguity, but does not establish sufficient VRAM or compatibility of the entire model stack. Keep these additional checks separate when interpreting a successful device-selection test.
For multiple workers, add PID and command. The same local ordinal can represent different devices in two processes with different filters, which is expected behavior.
Related operational reading: vllm docker first launch; gpu containers vs vms.