In this article
Typical GPU performance cannot be described by the best trial alone. Retain every attempt, its order, median and spread. First define whether timing covers a GPU kernel, complete operation or end-to-end serving. Different boundaries answer different questions on the same device.
Typical GPU performance cannot be described by the best trial alone. Retain every attempt, its order, median and spread. First define whether timing covers a GPU kernel, complete operation or end-to-end serving. Different boundaries answer different questions on the same device.
Define one trial
Record workload, inputs, image version, driver, cache state and warmup rules. Each trial must represent the same useful work. If only the first attempt includes artifact loading, label it separately from steady processing. Preserve failed and incomplete attempts outside the successful-value distribution. Silently discarding failures improves appearances while weakening evidence about repeatability.
Check timing methodology
CUDA operations are often asynchronous relative to the CPU. A CPU timer around kernel submission may measure queueing rather than completion. NVIDIA documents synchronization and CUDA-event timing for relevant scopes. Select the method according to the question and retain start/end boundaries. Client end-to-end timing is useful too, but includes networking and preparation. It must not be labeled pure GPU speed simply because the operation eventually uses an accelerator.
Calculate an illustrative median
For synthetic durations of 9, 10, 10, 11 and 20 milliseconds, the median is 10 and the mean is 12. The best value, 9, describes one attempt. The 20-millisecond observation needs explanation or investigation rather than automatic removal. These are not device benchmarks. Preserve sequence as well as sorted values: trial order can reveal warming, neighboring load or differences between cold and repeated startup.
| Report field | What it preserves |
|---|---|
| Run ID and order | Temporal structure |
| Median | Central value of selected trials |
| Spread | Differences among attempts |
| Failures | Conditions without useful completion |
Compare matched series
Use identical inputs and a declared repetition order for two configurations. Alternating or predefined randomization can reduce confounding from time drift, but preserve that design. Do not change driver, model revision and timer together. A median resists isolated extremes but does not fully describe latency tails. Show slow-attempt behavior separately when it matters rather than presenting median alone as a complete user-experience criterion.
Publish a reproducible artifact
Keep a run-level table with metadata and an explicit summary calculation. State attempt count, exclusions and reasons, and the workload scope of the conclusion. With few repetitions, avoid false precision from many decimal places. Store a new series alongside the original after environmental changes rather than overwriting it. The report then supports inspection of typical behavior and disputed trials without invented performance comparisons or a universal GPU ranking. Another operator should be able to recompute the summary directly from retained observations.
Related operational reading: inference benchmark method; rtx 5090 tokens per second.