In this article
Bound vLLM backlog using the work accepted by the serving system: request count and pending prefill volume are different quantities.
Bound vLLM backlog using the work accepted by the serving system: request count and pending prefill volume are different quantities. Equal-sized queues of short questions and long documents can produce very different waits. Establish each limit’s semantics in the installed release, then test a controlled burst. This concerns vLLM scheduling and associated admission, separately from authentication, account quotas, or a general API gateway.
Track three separate quantities
Active sequences describe running work, queued requests count accumulated requests, and queued tokens represent a token backlog. The vLLM reference defines max-num-active-seqs, max-num-queued-reqs, and max-num-queued-tokens separately. Respect the documented accounting scope, including the API process and data-parallel configuration. Do not interpret every option as the length of one physical GPU queue, or assume that counted tokens already occupy KV blocks. These controls use different accounting models and answer different operational questions.
Recognize the limits of request counts
Two inputs with identical output limits can require substantially different prefill work. Count-only protection therefore leaves considerable variation in waiting time. A token backlog accounts for input work, but its estimate can be conservative. Do not replace its semantics with a homemade count of remaining tokens without examining the implementation. The optimization guide connects this to scheduler execution budgets, which determine how accepted work runs rather than how much work should be admitted in the first place.
Exercise a controlled burst
Prepare short questions and long documents, recording actual input-token lengths. Submit each group with the same request count. Record accepted requests, expected rejections, TTFT, and completed responses for every run. Deliberately cross the selected threshold in a test environment to verify documented rejection behavior, including 503 for the applicable limit. This is an experiment design, not a predicted rejection count: existing in-flight work and timing affect the outcome, so the input list size alone cannot establish it.
Attribute rejection to its layer
The client should distinguish admission rejection from a generation failure after acceptance. Retain the status and timestamped server log. Immediate automatic retries can refill the backlog and conceal the limit’s effect. Use a fixed arrival schedule without uncontrolled retries during diagnosis. Where an upstream gateway has independent restrictions, identify which layer rejected the request. Otherwise a vLLM adjustment may appear ineffective even though the rejected traffic never reached the engine and could not have exercised its scheduler-aware accounting.
Document the resulting policy
Choose limits from acceptable waiting time for the relevant workload rather than the maximum number of open connections. Save runtime version, input lengths, data-parallel settings, and client behavior. Document execution budgets separately from admission backlogs: enlarging one can change the TTFT/ITL tradeoff without establishing a predictable boundary for the other. Repeat the burst after changing models because prefill cost changes. A useful deliverable is a table of limiting layers, units, and observed rejection behavior, distinguishing scheduler tokens, memory allocation, upstream rate limits, and active sequences.
Related reading
Planning a GPU pool: when is the next server actually needed?; Throughput and latency: choosing a GPU operating point.