# GPU benchmark confidence intervals: what counts as an observation

Diagnose gPU benchmark confidence intervals: what counts as an observation. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/gpu-benchmark-confidence-interval/
Language: en

A confidence interval is meaningful only after defining an observation. Thousands of tokens in one response are not thousands of independent machine trials. Specify the mean being estimated, independent repetitions and the suitability of the statistical method first.

## Define the estimand

State whether the mean concerns complete-operation duration, throughput across independent windows or another predefined outcome. A confidence interval is not a range expected to contain almost all future requests; that concerns the observation distribution. An interval for a mean is also not automatically an interval for a median. Methods and assumptions differ. Preserve units, inputs and timing scope with every original observation.


[Official reference 1](https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm).

## Inspect independence

Tokens within one autoregressive response share a request, model state and execution conditions. Consecutive windows in a long experiment can also be correlated. Treating them as independent trials enlarges nominal sample size without equivalent new information. Select an independent experimental unit and retain grouping. For comparisons on one node, consider time drift, shared load and cache reuse that can connect several attempts.


[Official reference 2](https://www.itl.nist.gov/div898/handbook/eda/section3/eda351.htm).

## Read the formula literally

NIST gives a t interval for a mean as mean ± t × s / sqrt(n). Here s is sample standard deviation, n counts observations and t corresponds to the selected level and degrees of freedom. Appropriate statistical assumptions are required; the formula cannot repair biased sampling or dependent observations. Distributional suitability matters especially for small samples. A 95% level describes long-run method coverage under its conditions, not a probability assigned to the fixed mean inside one computed interval.

| Term | Required statement |
| --- | --- |
| Observation | Independent experimental unit |
| Estimate | Which mean is estimated |
| Method | Formula and assumptions |
| Coverage | Selected confidence level |

## Examine artificially enlarged n

Suppose a report contains 12 separate runs, each producing many tokens. When estimating mean run duration, n concerns those 12 observations rather than their total token count. This illustrates analysis units, not measured hardware performance. Grouping and correlation require an appropriate design and justified method instead of substituting a larger number. A narrower interval is not improved evidence when obtained by miscounting independent data.

## Preserve a checkable calculation

Retain run-level values, grouping, method, confidence level and conclusion limits. Show failures and exclusion rules. Do not infer superiority solely from visual overlap of two intervals without a chosen analysis of the difference. If independence or distributional assumptions are unsupported, state uncertainty and redesign the experiment. A transparent range with a clear observation unit is more useful than a precise number describing a different object from the claimed GPU performance. Preserve enough information for another reader to reproduce the calculation and challenge its assumptions.

Related operational reading: [inference benchmark method](/journal/inference-benchmark-method/); [stable diffusion gpu benchmark](/journal/stable-diffusion-gpu-benchmark/).