Use llama-bench to isolate prefill and generation regressions

To localize a llama.cpp regression, separate prompt processing from token generation with llama-bench first. Total API response time combines computation, queueing, and client work. Compare pp, tg, and pg with matching settings, context depth, and builds. Incomparable runs cannot reveal whether a model phase changed or whether the experiment simply executed different work under different external conditions.

Interpret measurement modes

The llama-bench README defines pp, tg, and combined pg modes. They answer different questions and should not collapse into one tokens-per-second number. Preserve exact prompt-processing, generation, and context-depth settings. Isolated tg near a shallow context does not characterize every long conversation. Before comparison, identify the phase being investigated and verify that the command exercises it rather than another mode inherited accidentally from an older report. Preserve the full invocation with the raw output.

Fix build and placement

The build guide describes backend and compiler choices affecting the executable. Comparing commits built with different optimizations does not isolate source changes. Record compiler, Toolkit, architecture targets, offload, and threads. If a new backend is the intended variable, say so explicitly. Do not attribute the entire effect to one commit when CUDA targets, CPU-native optimization, or GPU-layer count also changed. That experiment can still be useful, but it answers a different question and needs a different conclusion.

Create a regression record

For old and new builds, make pp, tg, and pg rows at two selected context depths. Choose depths for the task and preserve them in the manifest rather than presenting them as universal defaults. Record settings, repetitions, variation, and functional smoke results. A pp-only regression points toward input processing. A tg regression only at greater depth motivates inspecting the corresponding state-access and execution path, not blaming checkpoint-loading disk speed. Keep the phase distinction visible in the resulting table.

Repeat conditions, not impressions

Use preserved inputs and enough repetitions to show variation rather than one favorable value. Account for warmup, background work, and changing power limits where relevant. Do not mix cold startup with a phase benchmark on an already-loaded model. Repeat the original build after the new one when environmental drift is suspected. If the effect is unstable, report uncertainty rather than publishing a precise regression percentage from one accidental run. Establish the observation before offering a mechanism for it.

Validate serving separately

A phase benchmark localizes computation but does not replace an HTTP workload with batching and queues. Follow local checks with representative serving tests and state both scopes. Keep binary checksums, model revision, commands, raw results, and context settings. Better pp does not automatically reduce TTFT behind an overloaded API. The report should identify the changed phase, depth, and repeatability, helping developers reproduce it and operators assess relevance without treating one aggregate throughput figure as a universal property of the model, GPU, or complete server.

A GPU inference benchmark you can trust; vLLM bench serve: Read a Serving Benchmark Result.