In this article
A working nvidia-smi establishes utility/NVML access, not complete readiness for a CUDA application. NVIDIA container compute and utility capabilities inject different driver components. Check NVIDIA_DRIVER_CAPABILITIES alongside device visibility before reinstalling the entire software stack.
A working nvidia-smi establishes utility/NVML access, not complete readiness for a CUDA application. NVIDIA container compute and utility capabilities inject different driver components. Check NVIDIA_DRIVER_CAPABILITIES alongside device visibility before reinstalling the entire software stack.
Preserve two independent results
Record host nvidia-smi, container nvidia-smi and the exact failure of a small CUDA workload. Establish where each command runs and under which user. Compare within the same container rather than different images with different libraries. Management-tool availability and successful CUDA context creation are different functions. A model failure after CUDA initialization belongs to another stage and cannot be explained solely by utility capability.
Inspect the capability variable
NVIDIA_DRIVER_CAPABILITIES controls which driver libraries and binaries are mounted into the container. Utility supports nvidia-smi and NVML; compute supports CUDA and OpenCL. An explicit value replaces the capability set rather than appending to every previous setting. Preserve effective environment and runtime configuration. Do not confuse this variable with Linux capabilities or seccomp: the same word refers to a different mechanism and configuration scope.
Examine a bounded example
An illustrative container sets NVIDIA_DRIVER_CAPABILITIES=utility. nvidia-smi works, but a compute application lacks the required compute environment. A new test launch explicitly selecting compute,utility should repeat both checks. This demonstrates the distinction, not a repair for every CUDA error. Driver incompatibility, architecture, permissions and library conflicts can persist even with correctly declared capabilities and an available device.
| Layer | Fact to verify |
|---|---|
| Device injection | The intended GPU reaches the container |
| Utility | NVML and nvidia-smi are available |
| Compute | CUDA driver components are mounted |
| Application | A minimal workload actually executes |
Change the smallest setting
Declare only required driver features and create a new test container. All expands the component set but obscures what is necessary. For inference, understand the compute path and utility observability; graphics or video need their own purpose. Check whether a template overrides the variable after configuration. Preserve device selection too: correct libraries cannot provide a GPU that the runtime never injected.
Confirm the compute path
Execute a small agreed CUDA workload and verify its result before testing the application. Retain image digest, driver version, visibility and capabilities. If the minimal task succeeds while the model fails, investigate application dependencies and configuration next. The artifact should contain both results rather than one green nvidia-smi line. This prevents post-update verification from relying entirely on a management tool that can work while the compute environment remains unusable. Repeat the distinction whenever the launch template or base image changes.
Avoid manually adding libraries to just one comparison container. That changes the experiment and prevents attributing results solely to driver capabilities in the declared launch.
Related operational reading: vllm install compatibility; vllm docker first launch.