# Compare SGLang Radix eviction policies under hot-set churn

Choose a SGLang Radix-cache eviction policy from the pattern of prefix reuse. LRU, LFU, and SLRU can behave differently when a stable hot set encounters a burst of one-off inputs.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/sglang-radix-eviction-hotset/
Language: en

Choose a SGLang Radix-cache eviction policy from the pattern of prefix reuse. LRU, LFU, and SLRU can behave differently when a stable hot set encounters a burst of one-off inputs. Compare them with equal capacity and identical request sequences. Knowing that prefix caching helps repeated inputs does not determine which entries should survive memory pressure or how to detect lost reusable state.

## Define the eviction problem

A bounded cache may need to remove previous state to admit a new input. The [SGLang documentation](https://docs.sglang.io/docs/advanced_features/radix_eviction_policy) describes policies based on recency, frequency, and usage segments. Their names do not imply a universal performance ranking. Recent activity may matter most for one workload, while surviving a brief burst of unique requests matters for another. Establish which prefixes return and at what intervals before designing a test that creates genuine competition for the evaluated cache capacity.

## Specify the hot set

Create a small set of repeated long instructions and a separate collection of unique documents. Preserve exact token sequences because similar text need not form a shared prefix after templating. Keep role order and control markers fixed. Do not describe an entire conversation as reusable when its beginning changes between requests. Verify server lifecycle as well: restarting the process between series can explain changed hits independently of eviction policy. State these conditions in the experiment record rather than leaving them implicit.

## Exercise a reproducible sequence

Warm the repeated set, submit a burst of unique inputs, and then return to the hot prefixes. Run the same sequence for each policy from comparable initial conditions. Record repeated prefill, cache observations, and TTFT for returning requests. This is a proposed experiment, not an observed hit-rate table. If one-off inputs displace useful state, determine which policy protects it on this traffic. Do not extend the conclusion automatically to all document collections or to a workload with different reuse intervals.

## Connect caching to latency

The [tuning guide](https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning) helps interpret cache behavior alongside other workload conditions. A high hit indicator does not replace user-visible latency, particularly when decode dominates the answer. Preserve match lengths and actual prefill work where available, rather than counting only requests with any hit. A short match and reuse of nearly an entire document have different value. Observe concurrent waiting work too: queuing can obscure the benefit of retaining the hot state even when eviction behavior improves.

## Write a bounded recommendation

Keep policy, parameters, capacity, and request sequence. Show behavior before the burst, during it, and after the hot set returns. Do not change allocator budgets or models simultaneously, because that would stop isolating eviction. Repeat the test when document distributions change. Explain which reuse pattern the choice protects and retain regressions. An eviction policy creates neither extra VRAM nor matching tokens where inputs differ; it allocates limited reusable state among competing inputs. Acceptance should therefore observe displacement and subsequent recomputation rather than merely confirm that prefix caching was enabled.

## Related reading

[Prefix caching: when repeated context helps a GPU pool](/journal/prefix-caching/); [KV cache and GPU economics: accounting for active-request memory](/journal/kv-cache-economics/).