In this article
A model that handles a short request may still fail on context memory; record the failure phase and total sequence budget.
If an LLM loads and answers a short question but runs out of CUDA memory on a long document, investigate context state and input-processing buffers. The weights generally have not changed. Count tokens after the chat template, reserve output capacity and control concurrent requests.
The aim is a useful, measured context boundary rather than a claim based only on the model’s advertised maximum window.
Count the actual input
Preserve the tokenised request, including system instructions, conversation history and retrieved material. Character count or PDF page count does not directly establish the sequence length used by the runtime.
Use a synthetic or approved document with a known answer. Start from a short working control and increase the relevant input gradually. Keep model artifact, format and output limit fixed while investigating the input boundary.
Budget the generated answer
Context limits generally cover input and output together. As an arithmetic example, a selected 8,192-token boundary needs reconsideration when input is 7,600 tokens and allowed output is 1,200. These numbers do not describe a particular model or guarantee placement on a GPU.
| Record | Why it matters |
|---|---|
| Input after templating | Actual sequence entering the model |
| Allowed output | Potential continuation |
| Active sessions | Combined sequence state |
| Failure phase | Input processing or later generation |
| Free device memory | Actual starting resource |
Avoid shortening output arbitrarily if doing so prevents the task from completing. The memory boundary must remain useful to the application.
Separate cache and input processing
Hugging Face’s KV-cache documentation explains sequence-state memory. vLLM’s optimisation guidance also addresses runtime resource behaviour. A failure before the first token can involve a different peak from a failure after a long generated answer.
Record that phase explicitly. Do not assume every long-context failure has one cause or that clearing allocator cache removes the request’s live state.
Find a reproducible working limit
Test increasing input lengths with one session, then longer output separately. Add concurrency only after the baseline is stable. Preserve errors, peak memory and correctness at every step.
Repeat the original short control after a failure. If it also changes, examine residual work and environment state before interpreting the context result. Label any offload or distributed execution as another configuration.
A bounded document workload can identify an appropriate role for existing equipment without promising the largest theoretical context or arbitrary parallel access.
Continue with vLLM max-model-len: Set a Meaningful Context Limit and CUDA Out of Memory with Concurrent Requests.