In this article
Heterogeneous GPU results can be combined only with compatible units and an explicit workload mix. Preserve each device and request class. Averaging individual rates or percentages does not always describe total throughput, especially when durations and work volumes differ.
Heterogeneous GPU results can be combined only with compatible units and an explicit workload mix. Preserve each device and request class. Averaging individual rates or percentages does not always describe total throughput, especially when durations and work volumes differ.
Define the aggregate question
Specify whether the target is simultaneous pool capacity, typical device performance or completion of a fixed dataset. These require different aggregation. Retain device identity, model/format, input/output work, concurrency and success criteria. When devices serve different classes, differences may reflect assignment rather than hardware superiority. Do not merge short and long operations without labels merely because both are called inference requests.
Preserve per-device data
For each device/class pair, retain completed work, elapsed time, errors and trial order. Mean and median describe different distribution centers, as NIST explains. An aggregate without original rows hides its weights. Equal averaging gives two devices equal influence regardless of completed work. A pool-wide measure may need different weighting; attractive speed alone does not justify an artificial statistical weight.
Examine a shared time window
In a synthetic example, two workers run simultaneously for 60 seconds on one operation class. One completes 30 and the other 50, yielding 80 operations in the common window if results are not duplicated and success criteria match. This is arithmetic rather than a device benchmark. Sequential trials have a different experiment duration. Different classes also require an explicit mix before combined counts per minute can support a fair comparison.
| Report field | Required basis |
|---|---|
| Device/class | Matched definition of useful work |
| Common window | Known simultaneity |
| Completed/errors | Successes and failures retained |
| Aggregate weight | Explicit weighting rule |
Inspect changing mix
For comparisons across periods, hold class distribution constant or standardize using predefined weights. Otherwise apparent improvement can result solely from more short tasks. Do not average ratios without inspecting their numerators and denominators. Define independent trials before estimating uncertainty; many tokens from one operation do not create an independent device sample. State method and assumptions instead of inferring superiority from one rounded pool score.
Retain a reproducible summary
Preserve per-device observations, class definitions, time window, weights and aggregate formula. A reader should recompute the result and inspect each contribution. Keep failures and unavailability separate from successful-request speed. Recheck mix after routing changes: identical GPU inventory does not establish an identical workload. The artifact evaluates a particular heterogeneous pool while retaining device differences and measurement boundaries rather than inventing a universal ranking. It also allows an inconclusive comparison to remain visible instead of forcing every dataset into a single winner.
Related operational reading: mixed gpu fleet; rtx 3090 vs 4090 llm.