In this article
Reserved memory includes allocator-held capacity; its difference from allocated memory alone does not prove a leak or fragmentation.
Allocated and reserved in a CUDA OOM message describe different layers of process-memory accounting. Allocated relates to live tensor memory; reserved includes capacity held by the allocator, including reusable blocks. Do not add them as independent expenses. A large reserved figure alone does not prove a leak or fragmentation.
Start by identifying which tool or runtime produced the values and the point in execution they describe.
Establish the measurement source
A framework report, device-management tool and process monitor can show different boundaries. Preserve tool versions, units, timestamps and the complete relevant error. Avoid comparing two unlabeled screenshots as though they measure one quantity.
| Observation | Useful interpretation | Limit |
|---|---|---|
| Allocated | Live tensor-related use | Not all system activity |
| Reserved | Allocator-held memory | Overlaps allocated capacity |
| Device usage | Broader GPU observation | May include other processes |
| Free memory | Current availability | Changes with workload state |
Keep the host’s physical inventory separate from runtime accounting. Neither an allocator label nor a combined total creates additional device capacity.
Distinguish live data from cache
PyTorch’s CUDA memory guidance explains the caching allocator. In a diagnostic experiment, compare state after loading, after a request and after releasing your own result references. Preserve the lifecycle being tested.
Clearing unused cache does not unload live model weights. It also does not establish that a fundamentally oversized model will fit. Do not insert experimental cleanup behaviour into a working service without understanding object ownership.
Look for reproducible growth
Repeat an identical bounded request and record observations across several completions. Separate expected retained state, caches and still-running tasks from unexplained growth. A one-time high value is not by itself a leak diagnosis.
Return to the original control after changing settings. If the baseline changes, document the sequence that produced it. Other applications or pending work can affect the next run independently of the selected model.
Investigate fragmentation with evidence
Fragmentation concerns whether free blocks can satisfy an allocation, not simply the difference between reserved and allocated. Use memory snapshots and allocator options documented for the installed PyTorch version when that investigation is warranted.
vLLM also has runtime-specific cache, profiling and preemption behaviour. Start with its own logs and optimisation guidance rather than applying a universal environment variable from another allocator setup.
A useful report identifies stage, artifact, resource observations and repeat conditions; it does not declare hardware defective from two overlapping memory figures.
Continue with CUDA Out of Memory While Loading a Model: A Check Order and CUDA Out of Memory with Concurrent Requests.