In this article
Define useful serving capacity through completed requests meeting first-token and token-pacing limits, then keep that measure separate from raw throughput.
A server can generate more tokens per second while completing fewer requests within a useful response-time budget. Goodput asks how much work meets a defined service objective. For interactive LLM serving, this commonly includes a first-token limit and a generation-pacing limit applied to the same request, rather than separate attractive averages.
Define success before running the test
The DistServe paper motivates capacity under latency constraints. Write the actual acceptance rule for your service: which requests count, which timing boundaries apply, and whether successful completion is required. Different services may legitimately choose different rules.
For an interactive assistant, a fast first token followed by an unusable stream may fail the objective. For a batch extraction job, total completion deadline and correct output can matter more. Do not copy an interactive SLO into a batch report and call the resulting number universally useful capacity.
Apply joint conditions per request
Keep a record of each request’s completion status and relevant timing measurements. Count a request only when it meets all chosen conditions. Passing the first-token condition for one subset and the pacing condition for another does not imply that either subset meets the joint objective.
The vLLM serving benchmark reference exposes goodput options and metric definitions. Verify units and the selected release’s calculation. TPOT, individual token gaps and end-to-end latency are different quantities; a label should say exactly which one the criterion uses.
Preserve failures in the denominator
Record rejected, timed-out, cancelled and incomplete requests under a stated policy. Removing them silently can make a saturated service look efficient. Separate intentional client cancellation from service failure where the trace permits that distinction.
Also keep the output-length distribution visible. A request-rate figure and a token-rate figure answer different questions when outputs vary. Compare workload classes separately if one configuration completes mostly short tasks while another also completes the long tasks that the product needs.
Find a sustainable operating region
Increase offered work in controlled steps while keeping the trace and arrival model explicit. Observe the point where useful completions stop growing or tail latency breaks the objective. An all-at-once burst and a steady arrival process should not be reported as the same load.
The vLLM parallelism guide describes layouts whose tradeoffs require measurement. Compare them using equal hardware accounting and the same success rule. A higher aggregate decode rate does not compensate for a larger share of requests missing their deadline.
Publish both capacity and its contract
State the model revision, hardware topology, context and output distributions, arrival pattern, success rule and test duration. Present raw throughput alongside goodput, not as a substitute. Include enough timing detail to explain why requests fail the objective.
Goodput is not a claim about answer truth unless quality is explicitly part of the acceptance rule. It is a defined useful-work measure. Keeping that boundary clear produces an infrastructure result that another operator can reproduce without confusing an engine’s busiest moment with a reliable service capacity.
Related reading
Throughput and latency: choosing a GPU operating point, Inter-token gaps and throughput: do not average different quantities.