In this article
A protocol creates trust in a benchmark, rather than a large tokens-per-second number. Preserve conditions, separate modes and show errors and queues alongside results.
A technical report helps GPU owners decide whether to provide a machine to a pool. It helps infrastructure funders assess a proposed configuration. Both need reproducible results that can be checked after a model update or site move.
Describe what is measured
Record exact GPU, host, network, driver, engine and model versions. Save weight format, chat template and launch parameters separately. State whether you measure standalone execution or a network service.
vLLM provides benchmarking tools, including a serving benchmark for server tests. A tool name does not replace workload description: the same tester can use different inputs and modes.
Prepare representative traffic
Use several input/output length combinations. Save the requests or generation method. For configuration comparisons, the set must remain understandable and available without customer secrets.
Distinguish observed real traffic from a synthetic hypothesis. Testing a hypothesis is appropriate when future-user data is absent. Presenting a convenient synthetic set as confirmed demand structure is not.
Mark repeated context. Otherwise warm caching may silently explain one run’s advantage.
Separate modes over time
Measure startup, warmup and steady operation separately. Compare equivalent intervals. If one server is tested immediately after loading and another after several minutes, the difference cannot honestly be attributed only to GPU hardware.
Repeat the chosen mode and preserve variation. One favorable result does not show stability. Operating decisions need averages and notably slower requests.
Agree on no background tasks. If the server cannot be fully freed, record what occupied it and why that reflects actual operation.
Repeat the initial configuration after experiments. If its baseline changes too, check temperature, background processes and site conditions before comparing settings. This catches changing conditions otherwise misattributed to a new model.
Show inconvenient metrics
Include errors, cancellations, queues, first-token time and completion. Do not omit failed requests without explanation. Aggregate successful-response speed and failure share answer different questions.
Add whole-machine consumption and site conditions when comparing operating efficiency. GPU power alone excludes the rest of the server.
Keep raw results in an agreed format. A short table supports discussion, while underlying data allows checking conclusions without accessing equipment again.
Finish with a bounded conclusion
A useful conclusion is specific: this configuration handled this traffic under these settings. It does not promise monthly revenue or establish suitability for every model on the device.
Compare it with the pilot plan, GPU usage accounting and operating economics. Providing resources then becomes an informed decision rather than faith in an advertised figure.