Tune llama.cpp CPU threads separately for prefill and decode

Tune llama.cpp CPU threads for prompt processing and token generation separately. One shared limit can suit prefill while harming decode, or the reverse. Compare threads and threads-batch with a fixed build, weight placement, and inputs. High CPU utilization does not establish good latency: threads compete for cores, memory, and scheduling, so phase-level measurements matter more than maximizing worker count.

Separate the execution phases

The argument definitions assign threads-batch to batch and prompt processing, separately from the main thread control. Do not equate a vCPU with a physical core or infer an optimum from logical-processor count. Preserve actual process or container CPU allocation and environment limits. If the model is partially offloaded, fix that placement too. Changing which operations remain on the CPU changes the tuning problem and can invalidate comparisons made under another layer distribution.

Measure phases independently

llama-bench distinguishes pp, tg, and pg measurements. Use these observations to identify where a setting helps. Total HTTP response time includes additional components and cannot replace phase diagnosis. Retain prompt length, context depth, and generated-token count. Short decoding near an empty context does not automatically represent a long conversation, whose state access and memory behavior can respond differently to thread count. Keep the tested context conditions with each result rather than treating model identity as sufficient description.

Construct a thread matrix

Create rows for permitted threads values and columns for threads-batch, staying within the agreed CPU allocation. Record pp, tg, pg, and functional correctness in each cell. This is an experiment design without predetermined winning numbers. Preserve both effects when a setting improves prompt processing but slows generation. Choose according to the target workload’s phase balance, rather than selecting a single attractive throughput result while hiding the cost imposed on the remaining request path.

Control affinity and contention

Keep CPU masks or ranges, NUMA placement, and consequential neighboring workloads. Changing affinity with thread count makes attribution ambiguous. Repeat the original configuration after the series to detect environmental drift. Full utilization of every logical processor is not an acceptance criterion; memory bandwidth can saturate earlier. If additional threads increase jitter, inspect contention and scheduling instead of assuming linear scaling with each available vCPU. A larger process count is a resource choice, not proof of more useful work.

Preserve the chosen profile

Record build, backend, offload, both thread controls, affinity, and fixture inputs. State whether the application prioritizes long-document intake or consistent decode. Repeat after CPU allocation, container-limit, or layer-placement changes. The conclusion should describe a useful setting range and observed regressions, not one universal thread count. Additional RAM cannot substitute for diagnosing compute and bandwidth behavior. The record should let another operator repeat phase measurements and understand why the selected combination matches the actual mix of prompt processing and generation without confusing virtual-processor labels with physical resources.

When to test CPU LLM inference before buying a GPU; NUMA in GPU servers: why resource placement matters.