Why temperature zero is not enough for deterministic SGLang output

Temperature zero in SGLang does not guarantee identical output when batch composition changes. Greedy selection depends on computed logits, which can be affected by numerical execution order. Compare the same request alone and alongside other requests. Before calling a difference a model bug, establish the supported deterministic path and the constraints of the selected backend, cache mode, and engine release.

Separate sampling from computation

Greedy sampling removes random distribution sampling, not every execution difference. The SGLang guide explains batch-dependent behavior and deterministic-inference features. One repeated answer cannot establish a bitwise guarantee. Preserve sampling settings, revisions, and environment. Distinguish identical input tokens from merely similar text: a tokenizer or template change introduces differences before numerical operations and requires a different investigation. Pin that input boundary before attributing output differences to the composition of concurrent requests.

Check the supported path

Validate deterministic behavior against the feature matrix for the particular release. The attention-backend description helps identify valid combinations. An enabled flag does not establish equal guarantees across all kernels and cache modes. Record the effective backend and prefix-reuse conditions. If the required combination is unsupported, state that limitation. Do not conceal it by switching backends without recording the change, because that also alters performance conditions and the scope of the behavior being tested.

Exercise batch invariance

Use a short question with preserved token IDs. Run it alone, in a series alongside other short questions, and beside a long input. Keep the generated sequence token by token and locate the first difference. This defines an experiment without predicting its result. If inputs differ, return to rendering. If they match, compare execution paths and supported deterministic settings before concluding that computation depends on batch composition. Keep arrival ordering and the other requests as reproducible fixtures too.

Choose an acceptance criterion

Some tasks require exact token sequences; others require equivalent structure or domain results. State the requirement before seeing differences. Text equality, token equality, and business validation answer different questions. Preserve raw tokens when exact matching matters. If semantic equivalence is acceptable, define a testable rule rather than relying on an operator’s impression. Measure the selected mode’s resource and latency cost independently instead of assuming deterministic execution is free. A successful correctness fixture is not itself a performance measurement.

Preserve the guarantee’s scope

The report includes engine version, model commit, tokenizer, backend, cache conditions, and batch shapes. Repeat after consequential component updates. Do not extend one machine’s result to every GPU architecture or platform. A reproducible difference should include a minimal fixture and first differing position. Separate input variation, sampling, and numerical execution so that “determinism” does not hide three distinct problems. Restricting execution paths may be a valid operational decision, but it must follow documented support and observed repeats rather than the assumption that temperature zero disables every source of variation.

Continuous batching: why concurrent requests change GPU value; Reproducible inference stacks: preserving a GPU pilot’s result.