Tune vLLM chunked prefill token budgets for mixed traffic

Tune the vLLM chunked-prefill token budget against a mixture of new long prompts and requests already generating answers. Fast first-token delivery and consistent subsequent-token intervals are different objectives. Change max_num_batched_tokens on a fixed workload while measuring TTFT and ITL separately. An aggregate throughput number cannot reveal which request group experienced a regression after the adjustment.

Understand the scheduler step

Chunked prefill processes a long input in pieces rather than as one large operation. The vLLM optimization guide describes decode prioritization and the token-budget latency tradeoff. This budget concerns work scheduled in a step, not the maximum context of an individual request. Increasing it is therefore not a way to permit a longer conversation. Establish the operational objective first: improve newly arriving long requests or protect the token-delivery intervals of answers already in progress.

Hold other constraints fixed

Preserve sequence limits, context limits, output settings, and cache conditions. Check current scheduler controls in the serve reference. Simultaneously increasing concurrency, output length, and the token budget prevents a meaningful attribution to chunked prefill. Record the input mix as well: short questions and long documents create different conditions. A control run must use the same text and stopping rules, otherwise the actual amount of computational work changes independently of the scheduler configuration.

Build a mixed-traffic experiment

Prepare one group of short conversational requests and another containing long documents. Submit them together using a saved arrival schedule rather than manually issuing isolated calls. For every configuration, create columns for short-request TTFT, long-request TTFT, active-answer ITL, and total completed throughput. Populate them only with measured results. This grouping can reveal a case where overall throughput barely changes while long prefills harm short-dialogue responsiveness, or where protecting decode increases the waiting time for document-heavy requests.

Interpret the observed tradeoff

If long-request TTFT worsens while ITL becomes steadier, the budget may be protecting decode more strongly. If long inputs finish prefill sooner while active-answer intervals increase, inspect how much prefill enters scheduler steps. Confirm these interpretations with profiling rather than deriving them from one statistic. Examine distribution tails: an occasional large input can cause an important regression that disappears in the average. Low utilization on one GPU is not sufficient evidence that every relevant scheduling limit should be raised.

Preserve the tuning decision

Keep the dataset, arrival schedule, configuration, and per-group results, including cold and warm series where both matter. Define acceptable user-facing compromises before selecting a configuration, and retain rejected alternatives. Treat the chosen budget as workload-specific rather than a universal model optimum. Repeat the experiment when document lengths or concurrent-answer proportions change. Separate scheduler execution budgets from admission backlogs in the report: one determines how accepted work runs, while the other bounds accumulated work. This avoids trying to cure a long waiting queue solely by enlarging a computation step and makes the changed mechanism explicit between runs.

Prefill and decode: the work your GPU performs during inference; Continuous batching: why concurrent requests change GPU value.