In this article
The ideal four-bit 32B estimate leaves an incomplete budget; actual formats and serving state determine whether a useful workload fits.
For exactly 32 billion parameters, 16-bit weights occupy about 64 billion bytes. An idealised four-bit representation gives sixteen billion bytes, but this does not establish that any 32B model is ready for service on 24GB VRAM. Format overhead and request memory need separate checks.
Use the calculation as an initial filter, then test a specific supported artifact with a workload that is useful to the intended user.
Identify the model and derivative
The Qwen3-32B card describes a specific original model. A quantised derivative is another artifact with its own provenance and requirements. Preserve the revision, conversion method and loading path.
If a guide refers to a different parameter count or a mixture-of-experts architecture, correct the comparison first. Similar names do not justify transferring a dense 32B estimate to a different stored model.
Keep the arithmetic explicit
| Assumed precision | Only 32 billion parameters |
|---|---|
| 16 bits | 64 GB |
| 8 bits | 32 GB |
| 4 bits | 16 GB |
These decimal figures assume one representation for every parameter. They exclude scales, metadata, buffers and components stored at another precision. A download size and framework allocation can therefore differ without either being a complete service estimate.
Verify quantisation compatibility
vLLM documents support for quantisation methods and hardware. Match the artifact to a supported execution path. A flag does not convert arbitrary weights into every requested format.
Keep the environment reproducible and record errors before changing packages. If the selected artifact does not load, investigate the format and initial allocation rather than immediately reducing only the client prompt.
Measure request headroom
After loading, run one short control request with a known correct outcome. Then extend input and output separately and add concurrency last. Record per-stage memory, failures and useful quality. Successful loading is a narrower result than handling a long document.
Check other applications and free memory before the run. CPU offload, if supported, is a distinct hybrid configuration with its own timing. It is not proof that host RAM has become extra VRAM.
If the model fits only with an impractical input limit or unacceptable quality, compare a smaller candidate rather than forcing the original size. The hardware’s potential role depends on useful execution rather than the arithmetic alone.
Continue with Local LLMs on 24GB VRAM: Test More Than the Weights and CUDA Out of Memory While Loading a Model: A Check Order.