In this article
Overload status, waiting advice and retry policy need explicit semantics. Retry-After does not guarantee capacity will be available at that time.
A provider benefits from rejecting inadmissible load clearly rather than opening SSE and retaining work it cannot serve. But 429 alone does not describe the policy. Clients need to distinguish rate restrictions, temporary resource shortage and other conditions, while servers need evidence that rejection preceded expensive work. This is a provider-side contract rather than guidance for bypassing a user limit.
Name the rejection cause
RFC 6585 defines 429 Too Many Requests and optional Retry-After. It does not prescribe one quota design for all services. Describe the scope: key, tenant, route or another resource. Do not expose internal infrastructure details unnecessary for client decisions.
Rate policy and temporary capacity shortage may need different responses. Preserve an internal diagnostic reason and a documented external classification. A generic status without context makes unsuitable load hard to distinguish from a case where waiting helps.
Define Retry-After
RFC 9110 describes date and delay-seconds forms. Check that clients handle the advertised representation. Wrong units or date handling can trigger immediate repeated attempts rather than intended waiting.
The header does not reserve capacity or guarantee the next attempt succeeds. Applications need total-duration and attempt-count bounds. Where repeating means running generation again, acceptance ambiguity and original-operation semantics require separate treatment.
Test the admission gate
NGINX limit_req describes rate limiting with burst, delay and configurable status. It is one possible gate rather than a universal inference-queue model. Save actual configuration and the request-counting point before interpreting results.
One fixture uses low frequency; another generates a predetermined bounded burst. Both use short synthetic inputs and finite duration. Observe accepted, delayed and rejected operations separately. One error total hides whether queues grow or early admission limits work.
Exercise client behavior
A fake endpoint can return a bounded rejection sequence followed by an allowed response. This validates header parsing and retry budgets without overloading a GPU. Preserve attempt timestamps and the scenario’s stopping reason. Correctly exhausting the budget is also a completed test outcome.
Add rejection without Retry-After and a malformed header value. Clients need documented safe behavior rather than indefinite waiting or a tight loop. Fallback rules belong to the application; HTTP does not choose them for the operator.
Do not turn rejection tests into an SLA
The check establishes a known admission policy for one version, not guaranteed traffic capacity. It also does not mean external routers retry identically. After gate changes, repeat fixtures and reason reconciliation.
Related guides cover client-side 429 and bounded fallback. The focus here is different: the service’s honest overload response, consistent with observations before streaming and controlled retries. Preserve whether rejected work entered the engine, where that is observable, so the test can separate a cheap admission refusal from an expensive operation that merely failed late.