# Prefill and decode: the work your GPU performs during inference

Why long-input processing and response generation stress a server differently, and how GPU owners can choose a useful technical-pilot workload.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/llm-prefill-decode/
Language: en

Server owners often see inference as one operation: a request arrives, the GPU works and an answer appears. Workload selection benefits from separating input-context processing from generation of new tokens. This reveals where time goes and which tasks existing hardware should test.

## Two stages of one answer

Input-sequence processing is usually called prefill; subsequent response generation is decode. The [vLLM optimization documentation](https://docs.vllm.ai/en/stable/configuration/optimization/) considers these stages separately when configuring request scheduling. Owners should test different workload shapes rather than seek one universal performance number.

Extracting a short conclusion from a long document and writing a long text from a short instruction can have similar total token counts but require different serving configurations.

## Build a matrix rather than one request

Offer four synthetic pilot profiles.

| Profile | Input | Output | What it checks |
| --- | --- | --- | --- |
| Short conversation | Short | Short | Basic responsiveness |
| Document analysis | Long | Short | Input processing |
| Text generation | Short | Long | Sustained output |
| Working session | Long | Long | Combined limits |

Start with one request, then gradually add concurrency. Record actual tokenized lengths: equal character counts do not guarantee equal work across models.

## Separate waiting from computation

Time to first token contains more than one model operation. Clients may wait for a queue, input processing and data transfer. “Slow to start answering” therefore does not necessarily mean a new GPU is needed.

Record request submission, first token and completion times, alongside queue state. This reveals whether delay comes from long input, overall load or an external connection.

For owners, this reduces unnecessary upgrades. Before buying a card, changing concurrent-request limits or serving different profiles separately may be worth trying.

## Test mixed traffic

Next run short and long tasks together. Otherwise a server can perform well on each class separately but poorly when combined.

A useful proposed criterion is that short requests retain agreed responsiveness while long work completes within acceptable limits. Choose specific boundaries for the product, rather than another company’s promotional benchmark.

Save model and environment versions, context limits, traffic mix and machine consumption. Without them, comparisons between days are unreliable.

## Use the result for onboarding

The matrix can identify a server role: short interactive requests, documents or another verified scenario. This is technical evidence for a pool discussion, rather than proof of future token sales.

State GPU, VRAM and expected availability, and use this matrix for the first discussion. Continue with [model memory](/journal/vram-and-models/) and the [pilot plan](/journal/pilot-checklist/).