In this article
Read the KV Block Manager as a memory hierarchy, including block movement, pinned host capacity, storage latency and version-specific integration limits.
The Dynamo KV Block Manager, or KVBM, manages inference state across a hierarchy rather than treating all available bytes as one pool. A block in GPU memory, host RAM or storage has a different access cost and lifetime. Planning the hierarchy requires more than summing advertised capacity across those devices.
Draw the block lifecycle
The KVBM overview describes allocation and management of KV blocks across heterogeneous resources. For a concrete deployment, trace allocation, completion, retention, offload, reload and eviction. Mark which component makes each decision and which process owns the data at that point.
A request that needs a block now cannot use an abstract total capacity figure. It needs a compatible block to arrive before its latency budget is exhausted. Label state waiting to move separately from state already resident and ready for an inference worker.
Assign a budget to each tier
The pinned Dynamo v1.5.0 configuration reference distinguishes tier-related settings, including host and storage resources. Use that reference as a versioned description, not as a claim that every tier is enabled or suitable in every later installation.
Reserve headroom in host memory for the engine, operating system, buffers and other processes. Pinned memory also has operational consequences beyond its size. A large storage tier does not justify a small or oversubscribed host staging area when transfers must pass through that path.
Measure the path between tiers
Characterize the transfer route used by the selected hardware and connector. Test it while inference runs, with the intended number of workers. An isolated storage benchmark cannot reveal contention with GPU copies, rank coordination or concurrent reloads.
Record bytes requested, bytes actually transferred, load latency and eviction pressure. Include a document whose blocks are spread across tiers. Counting that request as one cache hit conceals the cost of reconstructing its usable state. Capacity and access time should remain separate columns in the report.
Avoid stacking unrelated offload recipes
The Dynamo native vLLM offloading documentation describes another integration path. KVBM, engine-native offloading and an external caching connector are not interchangeable settings that can automatically be accumulated for extra capacity.
Pin the chosen manager and supported backend combination. Confirm who controls retention, transfer and block registration. Two components each assuming ownership of the same lifecycle can produce a configuration that starts successfully but behaves unpredictably under pressure or after a worker restart.
Evaluate a hierarchy under churn
Use a workload containing a stable hot set, one-off scans and bursts of revisited long contexts. Observe how eviction affects the hot set after a scan. Also restart a worker and remove access to an optional storage tier in a controlled environment.
Measure useful request completions and first-token tails, not only retained bytes. Define the ordinary-computation fallback for unavailable state and document the data-retention policy of each enabled tier. A disk or object-storage cache is persistent handling; it must not be enabled by accident in a service whose policy permits only temporary memory processing.
Related reading
Pinned GPU host memory: setting a page-locked buffer budget, KV cache and GPU economics: accounting for active-request memory.