In this article
The quantization flag is only one part of a supported artifact-to-hardware combination; it does not convert arbitrary weights automatically.
Launching a quantised model in vLLM requires agreement between the prepared checkpoint, quantisation method, GPU architecture and runtime support. The --quantization argument alone is insufficient for arbitrary weights. Begin with the artifact card and current compatibility guidance, then test memory and useful quality.
The outcome should identify one verified combination rather than treating all four-bit models as interchangeable.
Establish what was downloaded
Record original model, conversion repository, revision and selected files. A repository can contain several artifacts with different representations. A familiar model name does not identify which one the process actually loads.
Preserve the conversion’s documented requirements and any relevant tokenizer configuration. If provenance or the intended method is unclear, resolve that question before changing the host’s software environment.
Match method and hardware support
vLLM’s quantisation guidance lists methods and hardware restrictions. Reading a file and using an optimised compute path are separate capabilities. Check the installed version rather than assuming every release behaves identically.
| Component | Evidence |
|---|---|
| Checkpoint | Exact artifact and revision |
| Quantisation | Documented method |
| GPU | Supported architecture |
| Runtime | Version and loading path |
| Workload | Context, output and concurrency |
The syntax vllm serve CHECKPOINT --quantization METHOD shows argument placement only. Both placeholders require values appropriate to the artifact and installed serve reference.
Measure actual memory
For an illustrative fourteen billion parameters at four bits, a lower weight estimate is seven billion bytes. It is not a guarantee of running on a card with similar nominal capacity. Scales, metadata, tensors in other formats and runtime structures remain.
Measure loading, first request and the intended parallel profile separately. Record the stage of any error and actual free device memory. Do not treat host RAM as an automatic extension of VRAM.
Test meaningful outputs and preserve the combination
Use known-answer tasks, required structures or code tests. Compare formats on the same examples. Lower memory consumption is not sufficient if required correctness deteriorates or responses become incomplete.
Save checkpoint, method, GPU, vLLM version, context settings and quality results. Repeat control cases after an engine update because support and kernel selection can change. Preserve the original working baseline for comparison.
If only the card model is known, an initial discussion can define a bounded format investigation instead of promising universal support or requiring another purchase before testing.
Continue with A 14B Model on 16GB VRAM: Understand the Bit Estimate and CUDA OOM Allocated vs Reserved: Read the Evidence.