# SGLang HiCache vs RadixAttention: a hierarchy around prefix reuse

Separate radix-based prefix reuse from the host and storage hierarchy, then evaluate cache movement, model-state compatibility and working-set churn.

Author: Retrivium team
Updated: 2026-11-03
Canonical URL: https://retrivium.ai/journal/sglang-hicache-vs-radixattention/
Language: en

HiCache and RadixAttention are not two unrelated SGLang products competing to remember a conversation. RadixAttention concerns efficient reuse of shared prefix state. HiCache extends the memory hierarchy available around that reuse. Understanding their relationship is more useful than asking which name should replace the other in a launch command.

## Identify the prefix structure

The [SGLang paper](https://arxiv.org/abs/2312.07104) introduces RadixAttention as a KV reuse mechanism. Common beginnings in related requests can form reusable structure. The benefit depends on actual token prefixes, not merely two documents discussing the same topic.

Take a trace of multi-turn conversations and mark the portions repeated on later turns. Separate that from unrelated documents that only happen to have similar vocabulary. This shows the computation a prefix mechanism can potentially avoid before considering where the saved state is stored.

## Add the storage hierarchy

The upstream [HiCache documentation](https://github.com/sgl-project/sglang/blob/main/docs/docs/advanced_features/hicache.mdx) describes hierarchical caching and supported backend configuration. The important additional question is whether reusable state can remain useful beyond its immediate GPU residency and how it returns to execution.

Draw GPU, host and enabled external tiers separately. Label capacities, transfer routes and ownership. Storage beyond GPU memory can extend the useful lifetime of a context, but it does not make access equally fast. An offloaded hit is a different event from a directly resident hit.

## Check the model's actual state

The official [SGLang HiCache discussion](https://www.sglang.io/blog/hicache) addresses current model-state considerations, including hybrid architectures. Do not assume that every supported text model retains only conventional attention KV blocks or that one configuration applies unchanged to all architectures.

Record the exact engine revision, model, cache backend and supported state path. Keep model compatibility separate from storage connectivity. Successfully opening a backend does not demonstrate that the model's complete reusable state can be saved, restored and associated with the right prefix.

## Evaluate movement and churn

Use a stable hot set, returning long conversations and one-time scans. Compare GPU-only reuse with the selected hierarchy while keeping the request trace fixed. Include a quiet interval long enough for an offloaded context to return on a later request.

Record resident hits, host or storage hits, loaded bytes, recomputation and first-token tails. Inspect whether scans displace useful conversational state. A higher overall hit ratio can still worsen latency if many nominal hits must wait on a congested transfer path.

## Choose a supported retention boundary

Decide whether the workload needs only fast local prefix reuse, longer host-memory retention or an external shared tier. Weigh measurable benefit against cleanup, resource reservation and recovery work. Avoid enabling every tier simply because a backend option appears in documentation.

Describe the fallback for unavailable cached state and the sharing boundary between requests or tenants. Persistent external storage has different data-policy consequences from temporary GPU and host memory. In a provider deployment, state those boundaries explicitly and do not infer permission to retain content from the existence of a caching feature.

## Related reading

[Compare SGLang Radix eviction policies under hot-set churn](/journal/sglang-radix-eviction-hotset/), [Prefix caching: when repeated context helps a GPU pool](/journal/prefix-caching/).