In this article
Identify when allocation fails before changing settings; clearing allocator cache does not remove live model weights.
CUDA out of memory while loading a model means the process could not satisfy another device-memory allocation. Possible causes include weights, temporary conversion, other applications or runtime warm-up. Reducing client input helps only when the relevant allocations depend on that input.
Begin with the last successfully completed stage in the logs rather than changing several memory settings at once.
Build a startup timeline
Record process start, checkpoint reading, weight loading, cache preparation and API readiness. This separates slow storage from a VRAM allocation failure. A process existing is not the same as a ready service.
| Stage | Useful evidence |
|---|---|
| Before launch | Free memory and other processes |
| Checkpoint reading | Artifact and storage path |
| Weight placement | Format and device allocation |
| Runtime preparation | Profiling, buffers or graph work |
| Readiness | Successful control request |
Preserve the original error and software versions. Otherwise later changes can erase the evidence needed to understand the failure.
Estimate the weight boundary
Use actual parameter and format information for the chosen artifact. A simplified estimate is only a starting point and omits metadata and working structures. A flag labelled int4 does not transform arbitrary files into every supported quantised checkpoint.
If weights fundamentally exceed capacity, investigate a supported smaller format or model. Changing only a user’s short prompt will not unload those weights. Keep host RAM separate from device-local memory and label supported offload explicitly.
Check devices and competing processes
Confirm that the process uses the intended GPU. Record other applications and their ownership before considering any action. Do not stop unrelated work simply to make a demonstration pass.
Compare per-device observations for a multi-GPU launch. A combined free-memory total can conceal a failing device or uneven placement. Keep the host configuration and supported distribution method with the logs.
Investigate temporary peaks and allocator accounting
A model may fit after loading while conversion or runtime preparation creates a higher temporary peak. Match the failing stage to vLLM’s optimisation guidance and the selected hardware support.
PyTorch distinguishes allocated and reserved memory. They are not two independent physical capacities to add together. Clearing cache does not free live weights or solve fundamentally insufficient capacity. Use diagnostics appropriate to the installed runtime rather than an unexplained environment variable from another setup.
A bounded diagnostic record can identify a useful next test and avoids treating one failed artifact as evidence that the entire GPU has no practical role.
Continue with CUDA OOM Allocated vs Reserved: Read the Evidence and vLLM Quantization: Match the Weights, Format and GPU.