# vLLM KV cache preemption: why requests are recomputed

Understand vLLM KV cache preemption, distinguish it from OOM and compare concurrency settings using completed requests and end-to-end latency.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/vllm-kv-cache-preemption/
Language: en

vLLM preemption means the engine temporarily displaces active sequence work under resource pressure, including pressure on the KV cache. Depending on the engine mode and version, some work may need to be recomputed. The server can remain alive without a CUDA OOM while latency and useful throughput deteriorate.

## Distinguish preemption from a crash

An out-of-memory error usually interrupts an operation or process. Preemption is part of managing active work and is not necessarily a fatal event. Repeated warnings nevertheless deserve investigation, particularly when response times grow under an otherwise moderate request rate.

Record how often the events occur and what was running at the time. A single burst differs from sustained pressure. Keep the number of active sequences and actual input and output lengths alongside the log, rather than drawing a conclusion from one warning in isolation.

## Identify what consumes the cache

Cache requirements depend on model architecture, storage precision and the number of active tokens. Longer answers and more concurrent sequences can increase pressure even though the model weights have not changed. This explains why a single-request test may succeed on a GPU that struggles with a stream of users.

Inspect the documented metrics for your version. Do not assume that a larger configured context is free capacity. The full workload includes input tokens, generated tokens, concurrency and the memory needed by other buffers. Requested output allowances and actual completed output lengths should both appear in the report.

## Change one workload dimension

Keep the model and input documents fixed, then compare two concurrency levels. For each, record preemptions, completed requests, time to the first meaningful token and total response time. Next, test a shorter output allowance or shorter inputs separately.

If lower concurrency removes warnings, inspect the waiting queue. The delay may have moved from computation into queueing. An empty warning log is not sufficient evidence that users receive a better service. Measure the path from submission through the final token using the same criterion in both runs.

## Compare capacity changes deliberately

A larger engine memory budget may leave more room for cache, but only when VRAM and other allocations permit it. Alternatives include a smaller model, a supported precision change, another replica or distributed execution. Each has compatibility and quality implications.

Do not copy someone else's percentage as a universal setting. First identify the constrained resource, then measure the change. For batch processing, some extra latency may be acceptable; interactive applications can require a different operating boundary. Describe that boundary alongside the GPU specification when discussing a pilot.

## Turn the check into a practical next step

A live server can still spend too much work recomputing requests. Evaluate preemption alongside queueing, completed work and the latency users experience.

For the documented details, consult the [primary documentation](https://docs.vllm.ai/en/stable/configuration/optimization/); [configuration reference](https://docs.vllm.ai/en/stable/usage/metrics/). Continue with [vllm max model len](/journal/vllm-max-model-len/) and [vllm gpu memory utilization](/journal/vllm-gpu-memory-utilization/).