In this article
Validate a vLLM speculative-decoding draft and target as one configuration, rather than choosing a small model by size alone.
Validate a vLLM speculative-decoding draft and target as one configuration, rather than choosing a small model by size alone. Vocabulary, revisions, proposal settings, and verification cost all matter. A high accepted-token fraction does not by itself promise lower latency: additional computation can outweigh the benefit on a particular workload, especially when the number of concurrent requests changes.
Check vocabulary and method
The draft-model guide describes the usual shared-vocabulary requirement and a separate heterogeneous-vocabulary mode with its own sampling restrictions. Check the installed configuration schema before moving an example between releases. Different token IDs cannot be repaired by renaming a model. Pin the target, draft, and associated tokenizer artifacts, then establish that the selected method supports the intended combination. Merely accepting a checkpoint path without an immediate exception does not validate the complete draft-target contract.
Preserve proposal settings together
The speculative configuration schema defines the method, draft revision, and speculative-token controls. Save them together because changing the proposal count also changes target verification work. Do not treat older independent launch flags as universal syntax for a current runtime. Before benchmarking, retain the effective configuration and a minimal successful request. These artifacts distinguish failure to establish the pair from an unfavorable performance result on an otherwise working speculative path.
Compare two workload profiles
Create synthetic sets of short repetitive continuations and varied longer answers. For each set, compare the target alone with the same target using the draft, keeping prompts, sampling, and output limits fixed. Record acceptance, response duration, TTFT, and overall throughput. These are proposed measurement fields, not observed numbers. If acceptance is high but latency worsens, inspect draft and verification costs. If acceptance is low, investigate how well the draft matches the target and the task rather than increasing the proposal count blindly.
Account for resource cost
The draft consumes memory and performs computation. Its contribution should remain visible even when both models share one GPU. Record placement and available serving capacity when parallelism changes. Faster execution of one sequence does not establish better service for a queue of concurrent requests. Test a relevant concurrency range while preserving the same stopping rules. Different realized answer lengths can create a false improvement if the comparison considers only the elapsed time of two isolated responses rather than comparable generated work.
Define acceptance for the pair
Acceptance should establish that the intended method ran, returned the expected response structure, and did not silently take an unintended fallback path. Choose consequential metrics and resource constraints before inspecting results. Keep repository revisions, engine version, speculation parameters, and original requests. A heterogeneous-vocabulary sampling restriction is a specific reason to reject that combination, not a universal objection to draft models. State which workload was evaluated: findings about copied text or brief answers do not automatically cover extended reasoning. The deliverable is a reproducible pair specification and measurement protocol, not a promise that one switch accelerates every generation workload.
Related reading
A GPU inference benchmark you can trust; Throughput and latency: choosing a GPU operating point.