In this article
An overall deadline and a gap-between-reads timeout solve different problems. Tie budgets to one logical operation and observable stages.
Inference requests can involve connection, upstream, idle-read and total-operation timers. Calling them all timeout makes diagnosis imprecise. A server can keep sending service bytes while exceeding acceptable answer waiting, or a short idle timeout can interrupt work still within its total deadline. Define each constraint before testing it.
Name the budget’s start and end
RFC 9110 describes HTTP semantics rather than one application deadline policy for all LLM generations. A service must define whether total duration begins at acceptance, client submission or another documented boundary. Do not silently change the starting point to improve a metric.
Separate logical operations from attempts. With retries, user waiting can exceed each attempt’s timeout. Propagating remaining budget across layers is a separate design choice. Do not invent a universal header that every upstream provider supposedly understands.
Separate idle from total timeout
The NGINX proxy reference gives specific scopes for timeout settings. A limit between reads, for example, is not total response duration. Save the tested location’s settings and identify which event resets each timer. Useful content and service heartbeats can play different application roles.
If the source keeps emitting comments, transport waiting may continue after the product deadline expires. An overall bound must operate independently of service fragments. Otherwise an active connection can retain resources indefinitely.
Break down the request path
Record queueing, input preparation and output generation where those stages are observable. Do not infer hidden engine time by subtracting incomparable clocks on different hosts. Python time documents monotonic tools for measuring in-process durations.
Save acceptance, processing start, first useful event and completion. If only client timing is available, say so. It measures user experience but cannot establish exact internal GPU-time allocation. Report detail should match actual visibility.
Test delays separately
A fake upstream can delay headers, first content or the next event. Give each scenario a finite duration and expected classification. AbortSignal.timeout illustrates one client cancellation mechanism, but its active-time semantics should not automatically be equated with a server wall-clock deadline.
Use small fixtures without real user data. After ending client waiting, separately inspect upstream termination where observable. A closed reader does not prove complete server-chain cleanup or define accounting for performed work.
Keep the conclusion explainable
Name the expired budget, observation point and logical-operation outcome. “Request timed out” is too broad to guide configuration. Distinguish queue rejection, total-deadline exhaustion and gaps between reads so every parameter is not changed simultaneously.
Related topics cover prefill and decode and throughput versus latency. They support a specific waiting budget without universal thresholds, invented SLAs or claims of completed integration. Preserve each timer’s reset rule with the fixture; an apparently harmless change from useful-event time to any-byte time can otherwise invalidate a previous result while leaving parameter names unchanged.