vLLM bench serve: Read a Serving Benchmark Result

vllm bench serve tests an already running server with a defined request stream. Its result needs the model, input and output lengths, request count, arrival pattern and warm-up state. One tokens-per-second number does not explain an individual user’s wait for an answer.

Verify the endpoint before generating load so a routing or template error is not misreported as poor GPU performance.

Confirm the actual service

Request /v1/models and run one short generation. Check that the benchmark reaches the intended process and uses its API model identifier. Preserve the checkpoint, alias and address separately.

The serving benchmark reference defines the installed command’s arguments. Runtime metrics provide another observation source. Keep versioned names rather than assuming an old dashboard matches the current server exactly.

Define a small teaching workload

This example specifies an experiment, not expected card performance:

vllm bench serve   --backend vllm   --base-url http://127.0.0.1:8000   --model Qwen/Qwen3-0.6B   --dataset-name random   --random-input-len 512   --random-output-len 128   --num-prompts 40   --request-rate 2

If the server uses an alias, supply the relevant --served-model-name while retaining the original model for tokenisation as required by the command. Random tokens help control length; they do not validate document understanding, coding quality or client-interface behaviour.

Relate throughput to user waiting

Observation	Question answered
Aggregate output rate	How much successful output the stream produced
First-token delay	How long a request waited for output
Completion time	When the answer finished
Errors and cancellations	Which tasks did not complete
Input/output counts	What work was actually measured

Preserve actual response lengths. High total generation can coexist with an unacceptable wait for one interactive user. Failures should remain visible beside throughput rather than being discarded from the report.

Separate warm-up and the main run

Record whether weights and caches were already prepared. Keep shared-prefix conditions explicit and compare similar states. Also measure normal restart readiness separately when recovery matters to the operating schedule.

Save GPU, host, model, weight format, image, server arguments and full benchmark command. Label changes of model or format as new configurations rather than a hardware-only comparison.

The report can support a bounded pilot or funding discussion without converting a synthetic token rate into expected paid utilisation.

Continue with vLLM served-model-name: Fix a Missing API Model and CUDA Out of Memory with Concurrent Requests.

Official references

• vLLM — serve
• vLLM — metrics