In this article
Check SGLang FP8 or FP4 compatibility across the checkpoint, scaling recipe, GPU architecture, and GEMM backend.
Check SGLang FP8 or FP4 compatibility across the checkpoint, scaling recipe, GPU architecture, and GEMM backend. The format name alone is insufficient. Similar weights can have several runtime execution paths. Establish which path is actually selected before measuring speed. Validate attention separately: its backend matrix does not replace compatibility checks for operations on quantized weights.
Start from the artifact
An offline-quantized checkpoint describes its method and weight representation. The SGLang quantization guide explains relevant metadata and GEMM alternatives. Do not add a quantization flag simply because another example uses it; the runtime may already identify the format from the artifact. Preserve configuration and revision, then compare them with the chosen implementation’s requirements. A conflicting override does not automatically convert a checkpoint into another quantization recipe. Establish the representation before tuning its execution.
Check hardware and dependencies
Match compute capability, scaling representation, and library or JIT requirements for the selected GEMM. Available VRAM does not establish that a compatible kernel exists. Save engine and component versions. If the backend is unavailable, distinguish unsupported hardware from a missing build dependency. These require different responses: one combination may be invalid in the evaluated release, while another needs an officially supported component installed correctly. Neither case should be diagnosed solely from model size.
Separate GEMM from attention
The attention-backend documentation describes an independent implementation choice. Successful attention execution does not validate quantized matrix multiplication, or vice versa. Create separate launch-record rows for weight GEMM and attention, showing requested implementation, selected implementation, and compatibility evidence. Add a third row for quantized KV where relevant. This prevents one repaired layer from being described as complete support for every low-precision operation used by the model.
Build a diagnostic record
For one checkpoint, record quantization metadata, SM, GEMM selection, attention selection, and a short functional response. If needed, test an unsuitable explicit choice only in isolation to inspect rejection behavior. Expect an understandable incompatibility report rather than a silent reinterpretation of weights. This record is not a throughput benchmark. Performance numbers require a separate series with fixed prompts, realized output lengths, and concurrency. Two recipes cannot be compared meaningfully from the shared label FP8 alone.
Accept upgrades explicitly
Repeat the record after runtime or GPU-placement changes. Retain selected-kernel logs and response fixtures to distinguish incompatibility from quality regressions. Validate an alternative backend independently rather than assuming equivalence. State release-specific constraints without extending one supported architecture to every GPU from the same vendor. The deliverable traces metadata to the implementation actually executed. It explains why equal-VRAM nodes can behave differently and permits performance discussion only after correct execution is established, without promising that every lower-bit representation must be faster.
Related reading
LLM quantization: checking quality before choosing a GPU; vLLM Quantization: Match the Weights, Format and GPU.