In this article
A running container is not an accepted inference service; preserve one successful control request and the exact launch configuration.
A first vLLM Docker launch needs a compatible NVIDIA GPU and driver, container access to that device and a supported model. Start with a small model and a local API so installation problems remain separate from capacity limits. Seeing a container in a process list does not establish successful generation.
Use a bounded test environment and an agreed host window, especially when the machine also supports the owner’s work.
Check the accelerator before downloading weights
Confirm that the host sees the intended GPU, then verify access from the container environment. Record driver and device information alongside the engine log. A large model download will not solve a GPU that is visible only outside the container.
The official GPU installation and Docker guides define the supported route. Select a compatible image rather than assuming every combination of host driver and package works.
Start a minimal local example
This command illustrates a small experiment, not a universal hardware recommendation:
docker run --rm --gpus all -p 127.0.0.1:8000:8000 -v vllm-hf-cache:/root/.cache/huggingface --ipc=host vllm/vllm-openai:latest --model Qwen/Qwen3-0.6B --max-model-len 4096The API is bound to the local host. Review the container options and host conditions before use. For a reproducible record, preserve the exact image actually pulled rather than relying permanently on a moving latest tag. The 4,096-token setting bounds this example only.
Wait for engine readiness
After startup reports readiness, request /v1/models from the local endpoint. Use the returned model identifier in a short control generation. An accessible model list and a completed inference request are separate checks.
If listing works but generation fails, preserve the response and relevant logs. Repeatedly deleting the entire weight cache can obscure the original problem and add unnecessary preparation work.
| Check | Evidence |
|---|---|
| Container GPU access | Intended device visible |
| Engine startup | Readiness log |
| API identity | Returned model identifier |
| Generation | Useful control response |
| Restart | Repeatable readiness |
Preserve storage and the baseline
Keep weights, temporary state and results distinguishable. Record what persists after stopping or removing the container. Test the chosen arrangement rather than assuming every path inside the container is permanent.
Save image, model, command, readiness time and one successful answer before load testing. It supports a concrete pilot assessment without claiming that the first launch establishes production throughput.
Continue with Install vLLM: Check GPU, Driver and Package Compatibility and vLLM served-model-name: Fix a Missing API Model.