# Throughput and latency: choosing a GPU operating point

Why maximum token output may not deliver the best user experience, and how to compare GPU server configurations through throughput and latency together.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/throughput-latency-tradeoff/
Language: en

Two server settings can offer opposite advantages. One handles more aggregate work; another begins answering an individual sooner. Owners should understand the operator’s objective, or a disagreement about “poor speed” may actually concern different goals.

## Agree what speed means

Conversations often mix time to first token, response duration, one client’s output speed and total work across requests. Define each separately and retain its calculation method.

[vLLM metrics](https://docs.vllm.ai/en/stable/usage/metrics/) include different queue and request-time measurements. Negotiations do not require memorizing every counter. Understand the stage a metric covers and what lies outside it.

Fast generation after a long wait is not a fast answer for users. A large machine-wide token count does not imply the same speed in each session.

## Choose the user scenario

Interactive assistants, background processing and large-document preparation require different trade-offs. Define acceptable waiting for the specific task before comparing settings.

Do not set one universal threshold for every service. If the product is undefined, test several profiles and show each limitation. Owners then learn more about possible equipment roles.

For pool funders, this matrix clarifies which technical capabilities additional GPUs would purchase.

## Build a load curve

Increase arrival intensity gradually. Record successful work per interval, first-token waiting and total latency at each step. Keep averages and the slow portion of the distribution.

A useful experiment table contains “load,” “successful requests,” “typical latency,” “slow requests” and “errors.” Apply identical measurement rules to every row.

Find the region where added requests mainly increase queues. It shows where the configuration stops meeting the objective. Crashing a server is unnecessary to find a sensible operating limit.

## Compare technical options

After measurement, discuss concurrency limits, another model profile, separated traffic or an additional replica. Consider new equipment alongside these alternatives.

The largest aggregate number is not always the best operating strategy. If users interrupt slow answers, actual usefulness may be lower than expected. This is a product hypothesis to test, rather than a future-revenue calculation.

Do not transfer the selected point to another model automatically. Context and weight-format changes require comparison again.

For clarity, select two neighboring steps and show specific user sessions from each. These examples explain the trade-off to equipment owners better than one general graph with an undefined meaning of speed.

## What GPU owners receive

The result is a clear server role and agreed workload limits. This supports predictable operation and reasoned expansion discussions.

Also see [continuous batching](/journal/continuous-batching/), [benchmark methodology](/journal/inference-benchmark-method/) and [pool capacity planning](/journal/inference-capacity-planning/).