# LMCache vs vLLM prefix caching: choose the cache boundary

Compare engine-local prefix reuse with an external KV cache, including native vLLM offloading, cache lifetime and the cost of moving cached blocks.

Author: Retrivium team
Updated: 2026-11-03
Canonical URL: https://retrivium.ai/journal/lmcache-vs-vllm-prefix-caching/
Language: en

LMCache and vLLM automatic prefix caching answer different deployment questions. Prefix caching asks whether a worker can avoid recomputing the beginning of a request. An external cache also asks where reusable state can live after it leaves that worker's GPU. Decide which boundary matters before adding another service to a serving stack.

## Start with engine-local reuse

[vLLM documents automatic prefix caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) as reuse of KV blocks for matching input prefixes. The newly generated answer still needs decoding. A cache hit therefore does not imply that a long output becomes free or immediate.

Consider a support assistant with a stable instruction prefix and changing customer messages. First measure how often the deployed worker actually sees that shared prefix. If it remains resident and traffic already lands on the same worker, an additional storage layer might contribute little while adding operational work.

## Identify what an external layer changes

[LMCache MP mode](https://docs.lmcache.ai/mp/) separates the caching service from the inference process and describes multiple storage tiers. This changes ownership, movement and potential reuse of state. It does not make an incompatible model, connector or block layout compatible by itself.

Draw the lifetime of a cached document across request completion, GPU eviction, worker restart and rescheduling. Mark the events at which reuse disappears today. That diagram is more useful than comparing two cache names without a specific missing capability.

## Include the native offloading alternative

It is no longer accurate to frame the choice as GPU-only vLLM versus all offloading in LMCache. The official [vLLM KV offloading guide](https://github.com/vllm-project/vllm/blob/main/docs/features/kv_offloading_usage.md) describes a native connector for moving reusable state into secondary tiers. Availability and supported combinations depend on the selected release.

Compare three concrete configurations: local reuse alone, the documented native offload path, and an LMCache integration. Record engine versions, connectors, storage capacity and whether their features can be enabled together. A feature existing somewhere in a repository is insufficient evidence for your exact image.

## Measure movement against recomputation

Use a trace containing first visits, repeated documents, interrupted sessions and competing prefixes. Keep the request contents and arrival schedule constant. Separate resident hits, offloaded hits and misses instead of publishing one combined hit rate.

Record first-token latency, transfer time, queueing, CPU memory and completed requests. An offloaded hit can still lose to recomputation when transfers are slow or contention is high. Conversely, expensive repeated prefills can justify storage even when aggregate hit rate looks modest. Neither result is universal.

## Choose the smallest useful boundary

Prefer the configuration that solves an observed lifetime or placement problem with an acceptable maintenance burden. Account for connector upgrades, cache invalidation, tenant isolation and storage cleanup as well as GPU utilization. Memory outside the GPU remains a finite resource.

Document when a miss is expected and how a failed load falls back to ordinary computation. A useful cache makes a known workload more efficient while preserving the answer contract. It should not become a hidden prerequisite for serving a correct first request.

## Related reading

[Prefix caching: when repeated context helps a GPU pool](/journal/prefix-caching/), [vLLM KV cache preemption: why requests are recomputed](/journal/vllm-kv-cache-preemption/).