vLLM stop strings and EOS: diagnose premature response endings

Diagnose an early vLLM response ending by identifying the stopping mechanism: a stop string, an ID in stop_token_ids, and EOS serve different purposes. A larger context window cannot repair an answer ended by a valid delimiter. Capture the request, output, and completion reason, then reproduce the case while changing only one stopping condition at a time.

Distinguish strings from token IDs

A string condition concerns generated text, whereas a token ID belongs to a specific tokenizer vocabulary. A readable fragment may span several tokens, while a control marker may have a dedicated ID. vLLM SamplingParams represents these controls separately. Do not copy a numerical ID from another model. Identical marker spelling does not establish identical vocabulary assignments. Check the actual tokenizer revision and the valid ID range before attributing termination to the serving engine.

Inspect EOS and inherited configuration

EOS settings may come from model or generation configuration. ignore_eos changes part of termination behavior, but it is not a universal switch that disables every limit. Hugging Face documents EOS and stop strings as separate generation controls. List the active stopping sources before determining which one triggered. Changing the template, EOS, and output budget simultaneously makes the experiment difficult to interpret because a successful second run no longer isolates a particular cause.

Reproduce a delimiter collision

Suppose an application stops on the string END, while the user asks about a variable named BACKEND. This is an illustrative diagnostic scenario, not a claim about a particular tokenizer. Create an input deliberately requesting the delimiter as ordinary content, and compare it with a neutral question. Temporarily remove only the relevant stopping condition. If output resumes, refine the delimiter contract or use a boundary that the model’s template actually assigns to the end of an assistant turn.

Inspect the complete stream

Keep the sequence of streaming events, final completion reason, and assembled text. A missing stop marker in the client’s text does not prove that the condition was never matched; settings can affect whether it is included in the returned output. A connection ending is also different from model termination. Compare streaming and non-streaming requests using the same parameters. Acceptance must inspect the final event rather than merely the last nonempty text fragment, which may arrive before the protocol reports completion.

Build regression fixtures

Cover an ordinary answer, a literal delimiter mention, a permitted empty answer, and output reaching its configured budget. Record the expected termination reason and stop-text handling for each fixture. Preserve the tokenizer revision with any explicitly configured IDs. Do not permanently disable EOS merely to pass one example: generation may continue beyond the intended conversational role. Repeat the fixtures whenever the template changes, because turn boundaries can change without a public API change. The resulting artifact should explain each ending and show the settings actually effective on the server, rather than only those written in client code.

Related reading

Testing streaming responses from the vLLM API; Long-context LLMs: testing a server for document workloads.