In this article
Rate limits, active-request counts and engine queues bound different quantities. Align them with an observable waiting budget.
New work can arrive faster than a server completes it. A service that only queues requests may keep returning successful HTTP acceptance while useful user waiting grows. Admission control defines an earlier boundary: what work enters, how much may wait and when rejection is explicit. It differs from batching already admitted requests inside an engine.
Separate three controls
A rate gate bounds arrival intensity over an interval. An in-flight limit bounds concurrently accepted operations. An engine queue represents waiting inside model execution. They are related but not interchangeable: infrequent long requests can occupy many active slots despite low arrival frequency.
Google SRE’s load guidance treats traffic management as a combination of mechanisms rather than one universal solution. Name the resource and position of every control in an inference API. Client IP is not automatically the right unit when traffic arrives through one shared router.
Inspect the selected gate
NGINX limit_req documents rate limiting, bursts and delays. It is a concrete mechanism but does not independently know model-request cost or replace GPU-memory accounting. Preserve effective configuration and explain the problem it addresses.
A gate delaying excess work introduces another queue whose duration must fit the user’s overall budget. Immediate rejection needs documented classification. Both require observation; otherwise reports cannot distinguish early protection from slow internal failure.
Build a bounded burst
Measure a short control stream first. Then send a predetermined finite set of synthetic requests on a known schedule. Do not increase load indefinitely until failure. The objective is to observe admission boundaries and the fate of accepted work when the selected limit is reached.
Record submission, acceptance where visible, first useful event and final outcome per request. Show accepted, queued and rejected separately. Aggregate tokens per second does not describe waiting cost, particularly when clients cancel slow operations before completion.
Connect rejections with metrics
The vLLM metrics reference documents queue observations and admission rejections for its version. Check actual names and availability on the installed server. A metric in another version’s documentation must not automatically appear in promised monitoring without verification.
Compare the external reason with its observed layer. Acceptance by a gate followed by engine rejection differs from early refusal before execution. This link supports tuning without false precision or replacing unknown internal state with one generic error count.
Select an operating boundary
The deliverable is a tested admission policy for a profile, not permanent maximum hardware capacity. Context, model and generation-duration changes can require revision. Save the workload alongside results so a lighter test cannot masquerade as evidence for heavier traffic.
Related guides cover continuous batching and pool planning. They assess admitted work; admission control provides an honest entry boundary without a universal SLA or claims about a project’s existing infrastructure. Keep rejected fixtures as regression cases too, because accidental removal of a limit can look like a success-count improvement while degrading the latency of every accepted request.