Evaluate vLLM n-gram speculation without a draft model

vLLM n-gram speculation proposes continuations from matches within the prompt without loading a separate draft model. Evaluate it on tasks containing repeated or copied text. It does not guarantee an improvement for open-ended writing. Decide from end-to-end latency and serving behavior, comparing candidate-search overhead with the useful proposals actually available in the chosen request set.

Understand the candidate source

Previously seen input fragments provide possible continuations instead of additional model weights. The vLLM documentation demonstrates the ngram method through speculative-config. The target still verifies proposals; eliminating a draft checkpoint does not eliminate overhead. If current text has little resemblance to earlier fragments, useful candidates may be scarce. That is a workload characteristic rather than evidence of a broken engine or insufficient GPU memory. Keep this distinction clear when deciding which requests belong in an evaluation.

Specify search boundaries

The speculative configuration reference defines n-gram lookup and proposal-count settings. Preserve these with the runtime version and exact input. Do not copy settings from a separate draft-language-model example, which uses a different proposal mechanism. Change one lookup bound or candidate count at a time while holding sampling and termination fixed. A wider search can add work without finding more useful matches in the documents being processed. Its effect needs measurement rather than an assumption that larger settings are inherently better.

Compare document editing with new writing

Take a small document and request a few word replacements while preserving everything else. Beside it, prepare a request for a new story without a source document. These are diagnostic workload profiles, not predetermined acceleration results. Run each with speculation disabled and with the ngram method. Record input length, realized output, response time, and available candidate-acceptance observations. A benefit restricted to editing should not become a general claim that the model now serves every user and task faster.

Distinguish speculation from prefix caching

Prefix caching reuses computation for an existing input prefix. N-gram speculation proposes future output tokens that must be verified. Keep prefix-cache conditions comparable between runs, or explicitly separate cold and warm series. Otherwise a shorter prefill phase can be wrongly credited to speculation. Control actual answer length as well: preserving more source text changes the amount of decode work. Comparing total response time without accounting for that difference can turn a task change into an apparent performance improvement.

Report a bounded conclusion

Keep the dataset, revisions, lookup configuration, and separate results for both profiles. State which requests contain repetition and retain regressions rather than reporting only an average. If the method helps a particular editing operation, limit the conclusion to that operation. Validate task equivalence and quality independently of timing: a shortened summary cannot replace a required full document. Repeat fixtures after engine changes because configuration schemas and implementations evolve. The report should explain where prompt lookup earns its cost and where disabling it is the better measured choice. The method’s name alone cannot justify a universal acceleration factor.

Prefill and decode: the work your GPU performs during inference; A GPU inference benchmark you can trust.