Persistent KV cache invalidation: when a prefix stops being reusable

A persistent KV cache can outlive an inference worker. That is useful only while its blocks still represent the computation the new worker would perform. Identical visible text is insufficient evidence: tokenization, model weights, attention layout or request isolation can change without an obvious difference in a chat window.

Define the identity of cached state

Start with an explicit namespace that describes the deployed weight revision, tokenizer and special-token rules, relevant model configuration, storage format and rank layout. Record adapters or other request-level changes that alter the computed state. This is an operator’s compatibility inventory, not a claim that every caching product automatically hashes all these fields.

For example, the LMCache Redis documentation illustrates keys containing model and worker-layout information. A readable model name is not automatically a cryptographic fingerprint of the exact checkpoint behind that name. Inspect the implementation used by the release before trusting a reused identifier.

Separate text equality from token equality

The LMCache MP HTTP API operates on token identifiers and model/layout metadata. Two clients can send the same words while producing different token sequences through templates, special markers or tokenizer revisions. Conversely, token identifiers only have meaning within the matching model and tokenizer context.

Save the serialized input and tokenization configuration with a small diagnostic corpus. During an upgrade, compare the actual model input rather than only the user-visible message. Include empty turns, tool results, multilingual text and long conversations whose prefixes contain earlier generated content.

Make invalidation a release decision

Classify an update before routing requests to the new worker. A display-label change may leave state compatible; new weights under the same repository name may not. Precision or connector-layout changes also need a documented compatibility answer rather than an assumption.

Choose one of three outcomes: a new namespace, verified compatible reuse, or deliberate cache clearing. Keep this decision separate from rolling back the application image. Rolling back software without restoring the matching cache boundary can reintroduce an incompatibility that the image rollback was supposed to remove.

Test a misleading hit

vLLM’s prefix caching guide explains the reuse mechanism. Extend normal cache tests with a deliberately stale namespace in an isolated fixture. The aim is to prove that the deployment procedure prevents reuse across the incompatible boundary, not to measure how frequently stale state happens to produce plausible text.

Compare cached and uncached outputs under a fixed evaluation procedure, allowing for documented numerical nondeterminism. Include correctness checks that would detect a wrong document or instruction, not merely a successful response status. A non-crashing cache load is not a semantic validation.

Include ownership and cleanup

Namespace state by the intended sharing boundary. The fact that two tenants submit a common prefix does not itself authorize cross-tenant reuse or disclosure of cache metadata. Align the design with the service’s data handling rules.

Assign expiration and cleanup responsibility for old releases, including failed rollouts and abandoned experimental adapters. A persistent cache needs an inventory just as persistent model files do. Otherwise, capacity usage grows while no operator can confidently explain which blocks are still useful or eligible for reuse.

Related reading

Prefix caching: when repeated context helps a GPU pool, Manage vLLM torch.compile caches across releases.