# Testing streaming responses from the vLLM API

Verify vLLM streaming with an unbuffered client, distinguish events from visible tokens and identify proxy buffering before changing GPU settings.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/vllm-openai-streaming-test/
Language: en

To test vLLM streaming, send `stream: true` to the appropriate endpoint and confirm that the client receives events during generation. If the entire answer appears at the end, client or proxy buffering may be responsible. That observation alone does not show that the engine ignored streaming.

## Establish a non-streaming baseline

Read the model identifier from `/v1/models` and first confirm an ordinary generation request. Use a short message and a modest output limit. This separates address, model-name and chat-template failures from problems reading a stream.

API compatibility concerns the protocol. It does not turn the selected local checkpoint into another provider's model. Save the actual checkpoint, server alias, message format and installed version so the test can be repeated after a restart.

## Use a client that does not add buffering

For an already running local server, the source example is:

```bash
curl -N http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Name three colours."}],"max_tokens":64,"stream":true}'
```

Replace the model identifier with the one actually advertised. The example prompt asks for three colours; its original payload is retained here. `-N` disables curl output buffering. A very short answer may finish too quickly to make intermediate events visually obvious, so this is a protocol check rather than a benchmark.

## Parse events rather than one JSON document

A stream contains structured messages. Some events carry metadata or completion information rather than visible text. The client must handle that sequence instead of attempting to parse the whole connection as one ordinary JSON response.

Test completion, errors and cancellation separately. A user interface that keeps waiting after the final event may have a handler problem. After cancellation, verify that leftover work does not contaminate the next measurement or create unexpected background load.

## Locate the buffering layer

Repeat the same request directly and through the intended intermediary. If direct output arrives incrementally but the proxied route delivers everything together, inspect that layer's buffering and timeout behaviour before changing the GPU.

Record client-side arrival times for several meaningful fragments. Time to the first event, first content token and final answer can differ, especially for models with structured reasoning output. Keep the criterion consistent when comparing servers. Test realistic load separately after the transport path is understood; an early metadata event is not itself a fast user-visible answer.

## Turn the check into a practical next step

Protocol streaming, time to the first meaningful token and total generation time are separate measurements. Use the same definition across configurations.

For the documented details, consult the [primary documentation](https://docs.vllm.ai/en/stable/serving/online_serving/); [configuration reference](https://docs.vllm.ai/en/stable/cli/serve/). Continue with [vllm localhost container port](/journal/vllm-localhost-container-port/) and [vllm bench serve command](/journal/vllm-bench-serve-command/).