# Tune vLLM chunked prefill token budgets for mixed traffic

Tune the vLLM chunked-prefill token budget against a mixture of new long prompts and requests already generating answers.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/vllm-chunked-prefill-token-budget/
Language: en

Tune the vLLM chunked-prefill token budget against a mixture of new long prompts and requests already generating answers. Fast first-token delivery and consistent subsequent-token intervals are different objectives. Change `max_num_batched_tokens` on a fixed workload while measuring TTFT and ITL separately. An aggregate throughput number cannot reveal which request group experienced a regression after the adjustment.

## Understand the scheduler step

Chunked prefill processes a long input in pieces rather than as one large operation. The [vLLM optimization guide](https://docs.vllm.ai/en/stable/configuration/optimization/) describes decode prioritization and the token-budget latency tradeoff. This budget concerns work scheduled in a step, not the maximum context of an individual request. Increasing it is therefore not a way to permit a longer conversation. Establish the operational objective first: improve newly arriving long requests or protect the token-delivery intervals of answers already in progress.

## Hold other constraints fixed

Preserve sequence limits, context limits, output settings, and cache conditions. Check current scheduler controls in the [serve reference](https://docs.vllm.ai/en/stable/cli/serve/). Simultaneously increasing concurrency, output length, and the token budget prevents a meaningful attribution to chunked prefill. Record the input mix as well: short questions and long documents create different conditions. A control run must use the same text and stopping rules, otherwise the actual amount of computational work changes independently of the scheduler configuration.

## Build a mixed-traffic experiment

Prepare one group of short conversational requests and another containing long documents. Submit them together using a saved arrival schedule rather than manually issuing isolated calls. For every configuration, create columns for short-request TTFT, long-request TTFT, active-answer ITL, and total completed throughput. Populate them only with measured results. This grouping can reveal a case where overall throughput barely changes while long prefills harm short-dialogue responsiveness, or where protecting decode increases the waiting time for document-heavy requests.

## Interpret the observed tradeoff

If long-request TTFT worsens while ITL becomes steadier, the budget may be protecting decode more strongly. If long inputs finish prefill sooner while active-answer intervals increase, inspect how much prefill enters scheduler steps. Confirm these interpretations with profiling rather than deriving them from one statistic. Examine distribution tails: an occasional large input can cause an important regression that disappears in the average. Low utilization on one GPU is not sufficient evidence that every relevant scheduling limit should be raised.

## Preserve the tuning decision

Keep the dataset, arrival schedule, configuration, and per-group results, including cold and warm series where both matter. Define acceptable user-facing compromises before selecting a configuration, and retain rejected alternatives. Treat the chosen budget as workload-specific rather than a universal model optimum. Repeat the experiment when document lengths or concurrent-answer proportions change. Separate scheduler execution budgets from admission backlogs in the report: one determines how accepted work runs, while the other bounds accumulated work. This avoids trying to cure a long waiting queue solely by enlarging a computation step and makes the changed mechanism explicit between runs.

## Related reading

[Prefill and decode: the work your GPU performs during inference](/journal/llm-prefill-decode/); [Continuous batching: why concurrent requests change GPU value](/journal/continuous-batching/).