In this article
Evaluate separate prefill and decode workers through phase interference, KV transfer, replica balance and useful completions under latency constraints.
Prefill processes the input context; decode generates subsequent tokens. Their resource patterns and user-facing latency pressures differ. Disaggregating them assigns the phases to separate workers or pools and moves the state between them. This can reduce interference, but it also introduces a transfer boundary and another capacity-balancing problem.
Establish the problem in a combined worker
Measure how newly arriving long prompts affect streams that are already decoding. Keep prompt lengths, output budgets and arrival timing in the trace. A phase split is relevant when this interference is material, not merely because the model supports a distributed connector.
The DistServe paper studies the relationship between phase separation and requests meeting latency objectives. Its reported gains belong to its experiments. They should motivate a test, not become a throughput promise for different GPUs, networks or request distributions.
Read the selected vLLM integration
The current vLLM disaggregated prefilling guide labels the feature experimental. Select an actual supported connector and record the release, model architecture, topology and failure behavior. A diagram with one prefill box and one decode box leaves most operational decisions unspecified.
Trace where a request is admitted, where its KV state is registered and when decoding may start. Include ownership if a client cancels while transfer is in flight. A disconnected client should not leave state permanently held by two pools that each expect the other to finish cleanup.
Budget state movement explicitly
Calculate and measure transferred state from the actual context, precision and parallel layout. Model weight size is not the transfer size for each request. Long inputs can create substantial KV traffic even when the generated answer is short.
NIXL is a data-movement library used by some integrations; it does not eliminate transfer latency or make every transport equally suitable. Test the selected backend across the intended host and device paths. Wide-area bandwidth and round trips require a separate justification from a local cluster fabric.
Size both pools against a mixed trace
A prefill pool can become the bottleneck while decode GPUs wait, or the reverse. Replay bursts, quiet periods, short chats and long-document requests. Change the ratio of phase capacity while keeping total hardware accounting visible.
Compare useful completions, first-token latency, token pacing and transfer queues against a combined-worker baseline. Reserve room for transitions and failures. If only the prefill benchmark improves while completed requests meeting both latency limits decline, the service has not gained the capacity that users need.
Decide where separation is worthwhile
Retain a combined configuration when its simplicity and locality provide better overall behavior for the workload. Adopt separation when measured interference reduction exceeds movement and coordination costs over realistic traffic, including tail cases.
Document eligible hardware groups and supported context limits rather than describing every remote GPU as part of one tightly coupled execution pool. The architecture should make clear which nodes can exchange state efficiently and which locations instead serve independent requests. That boundary is especially important in a network combining data centers and smaller partner sites.
Related reading
Prefill and decode: the work your GPU performs during inference, Diagnose KV transfer stalls in SGLang prefill-decode disaggregation.