# vLLM bench serve: Read a Serving Benchmark Result

Run and interpret vLLM bench serve with a verified endpoint, explicit input/output lengths, request rate, warm-up state and useful latency measures.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/vllm-bench-serve-command/
Language: en

`vllm bench serve` tests an already running server with a defined request stream. Its result needs the model, input and output lengths, request count, arrival pattern and warm-up state. One tokens-per-second number does not explain an individual user's wait for an answer.

Verify the endpoint before generating load so a routing or template error is not misreported as poor GPU performance.

## Confirm the actual service

Request `/v1/models` and run one short generation. Check that the benchmark reaches the intended process and uses its API model identifier. Preserve the checkpoint, alias and address separately.

The serving benchmark reference defines the installed command's arguments. Runtime metrics provide another observation source. Keep versioned names rather than assuming an old dashboard matches the current server exactly.

## Define a small teaching workload

This example specifies an experiment, not expected card performance:

```sh
vllm bench serve   --backend vllm   --base-url http://127.0.0.1:8000   --model Qwen/Qwen3-0.6B   --dataset-name random   --random-input-len 512   --random-output-len 128   --num-prompts 40   --request-rate 2
```

If the server uses an alias, supply the relevant `--served-model-name` while retaining the original model for tokenisation as required by the command. Random tokens help control length; they do not validate document understanding, coding quality or client-interface behaviour.

## Relate throughput to user waiting

| Observation | Question answered |
| --- | --- |
| Aggregate output rate | How much successful output the stream produced |
| First-token delay | How long a request waited for output |
| Completion time | When the answer finished |
| Errors and cancellations | Which tasks did not complete |
| Input/output counts | What work was actually measured |

Preserve actual response lengths. High total generation can coexist with an unacceptable wait for one interactive user. Failures should remain visible beside throughput rather than being discarded from the report.

## Separate warm-up and the main run

Record whether weights and caches were already prepared. Keep shared-prefix conditions explicit and compare similar states. Also measure normal restart readiness separately when recovery matters to the operating schedule.

Save GPU, host, model, weight format, image, server arguments and full benchmark command. Label changes of model or format as new configurations rather than a hardware-only comparison.

The report can support a bounded pilot or funding discussion without converting a synthetic token rate into expected paid utilisation.

Continue with [vLLM served-model-name: Fix a Missing API Model](/journal/vllm-served-model-name/) and [CUDA Out of Memory with Concurrent Requests](/journal/cuda-oom-concurrent-requests/).

## Official references

- [vLLM — serve](https://docs.vllm.ai/en/stable/cli/bench/serve/)
- [vLLM — metrics](https://docs.vllm.ai/en/stable/usage/metrics/)