# CUDA Out of Memory with Concurrent Requests

Diagnose CUDA OOM under concurrent LLM requests by separating clients, queued work and active sequences, then measuring a controlled load progression.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/cuda-oom-concurrent-requests/
Language: en

CUDA out of memory with several LLM requests can involve the combined state of active sequences. Each request needs context-related data while the engine schedules batches. A GPU that answers one user does not automatically support an arbitrary number of simultaneous clients.

Preserve the single-request profile first, then increase concurrency independently. Otherwise longer documents and more active work become difficult to distinguish.

## Separate clients, queue and execution

Connected clients, waiting requests and actively executing sequences are different counts. Record which one the load generator controls and which the server reports. Queue growth can increase user delay even before a memory error appears.

Use the metric names from the installed vLLM version. Its metrics documentation defines observations; a dashboard's simplified label should not be treated as an exact account of every scheduling state.

## Prepare identical work

An illustrative experiment uses 2,000 input tokens and a 400-token output allowance. Test one, two and four simultaneous requests with the remaining settings fixed. These numbers specify an experiment, not a performance claim for any GPU.

| Load stage | Keep fixed | Observe |
| --- | --- | --- |
| One request | Model and token profile | Baseline correctness and memory |
| Two requests | Same task distribution | Delay and shared resource use |
| Four requests | Same limits | Errors and queue behaviour |
| Return to baseline | Original control | Recovery and remaining work |

Record actual output lengths as well as allowances. Different generation lengths can alter the comparison even with an identical configured maximum.

## Do not treat batching as free memory

vLLM's optimisation guidance discusses scheduling, cache capacity and preemption. Batching can change efficiency, but it does not eliminate sequence state or make every workload fit.

Apply settings appropriate to your installed version and model. Record recomputation or preemption observations where available. A large aggregate token rate can coexist with poor time to first token for individual users.

## Establish an application boundary

Use a defined intake limit, queue deadline, cancellation behaviour and user-visible failure outcome. Verify what happens when load exceeds the tested profile. Do not silently lower required quality or repeatedly retry without a bounded policy.

The report should include concurrency, input/output counts, errors, delay and per-device memory. Keep a successful recovery check after the load test. A known safe scope is more useful than claiming server capacity from one successful request.

Continue with [CUDA Out of Memory on Long Prompts: Check the Context](/journal/cuda-oom-long-context/) and [What vLLM gpu-memory-utilization Actually Limits](/journal/vllm-gpu-memory-utilization/).

## Official references

- [vLLM — optimization](https://docs.vllm.ai/en/stable/configuration/optimization/)
- [vLLM — metrics](https://docs.vllm.ai/en/stable/usage/metrics/)