# Inter-token gaps and throughput: do not average different quantities

Distinguish TPOT, SSE event gaps and aggregate throughput through measurement boundaries, zero- and one-token outputs, weighting and a reproducible example.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-intertoken-gap-throughput-metrics/
Language: en

A client receives bursts while the server reports steady aggregate tokens per second. Both observations may be correct because they measure different quantities. Providers should separate time after the first token, application-event gaps and total output across requests. Otherwise an attractive average can hide individual waiting or instrumentation errors on short answers.

## Define the observation unit

The [GenAI server conventions](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-metrics.md) describe time per output token after the first token and its server-side scope. This is not simply the gap between network reads. One read can contain several events, and one event may represent multiple tokens under the selected interface.

Name the token-count source and timing boundaries. Do not recover tokens from spaces or equate SSE-event count with native output volume without a documented relationship. An approximation can be useful separately but must remain labeled as such.

## Handle edge cases

A one-token response has no post-first-token interval. Zero output is also outside ordinary TPOT arithmetic. Define whether these observations are omitted, classified separately and included in summaries. Substituting zero can artificially improve an average.

Empty visible content is not necessarily zero total work. Reasoning or other supported details may affect output-count meaning. Where evidence is insufficient, retain unknown and its reason rather than calculate a convenient value from incomplete text.

## Separate client gaps

The [GenAI client conventions](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/client-inference.md) describe caller-side chunk timing. These observations help characterize streaming experience but include delivery and client processing. Buffering can produce a large gap despite steady engine execution.

Save event sequence numbers and complete-event timestamps. Compare direct and proxied paths using a synthetic stream with known pauses. Validate the measurement tool before attributing delay to the model. Client chunk latency is not pure decode speed without further evidence.

## Choose aggregation weights

An average of per-request TPOT and total decode duration divided by total post-first tokens use different weights. Short and long answers influence them differently. Name the method and keep it unchanged across configurations rather than choose whichever produces an attractive result.

The [Prometheus histogram guide](https://prometheus.io/docs/practices/histograms/) explains distribution storage and limitations of precomputed quantile aggregation. Preserve statistics appropriate to the question. One average does not expose long pauses, while throughput summed over different intervals is not each user's speed.

## Reconcile without substitution

A useful artifact has separate columns for server TPOT, client event gaps, input/output counts and aggregate completion rate over the same interval. Every quantity needs a definition and unit. This reveals a real trade-off rather than an argument over the single word speed.

Related guides cover [throughput and latency](/journal/throughput-latency-tradeoff/) and [prefill/decode](/journal/llm-prefill-decode/). The result is consistent observation semantics, without universal performance figures, claims of an operating service or promised paid token volume. Include the treatment of short responses in the saved method so changes in request mix cannot silently alter the interpretation of the reported metric.