In this article
The 70B weight formula is a lower arithmetic guide, not a complete GPU requirement or a purchase recommendation.
For an illustrative model with exactly 70 billion parameters, weights require about 140 billion bytes at 16 bits, 70 billion at eight bits and 35 billion at four bits. These are simplified weight estimates, not complete serving requirements. Runtime structures and request memory must be added.
Use the arithmetic to reject unrealistic expectations, then inspect the actual artifact and measure a bounded workload.
State the formula and units
The simplified calculation is parameter count multiplied by bits per parameter, divided by eight. It assumes all parameters use the same representation and omits artifact-specific details.
| Assumed format | Weight-only decimal GB |
|---|---|
| 16 bits | 140 |
| 8 bits | 70 |
| 4 bits | 35 |
GB and GiB differ. Converting units changes the number shown, not the physical memory. The rounded 70B model name is also not a substitute for its actual parameter count.
Inspect the quantised artifact
Hugging Face’s quantisation overview describes different methods and implementations. A four-bit label does not mean the complete artifact consists only of four-bit values without scales, metadata or tensors in another representation.
Compare the estimate with the selected files and their tensor composition, then with loaded memory. For a third-party conversion, preserve provenance and the original model reference. Do not describe one conversion’s result as a universal property of the model family.
Add sequence state
Hugging Face’s KV-cache documentation separates model weights from sequence state. Input length, generated output and active sessions can change service memory. Loading a model successfully therefore does not validate its maximum window or target concurrency.
Record host RAM separately. A supported offload arrangement requires its own execution path and latency measurement, rather than simply adding CPU and GPU capacity.
Check placement before choosing hardware
If one device cannot hold the required configuration, investigate supported model partitioning. Several GPUs do not automatically form one fast pool. Preserve connections, device count and measured usage per card; one device can run out before a combined total appears exhausted.
Compare the distributed option with a smaller model when the task permits it. Quality, acceptable delay and operating cost remain the decision criteria, not reaching a particular parameter count.
The final record should distinguish arithmetic, loaded memory and the tested request profile. A formula informs the next test; it does not guarantee acceptance, utilisation or a commercial outcome.
Continue with 48GB VRAM for LLMs: One GPU and Multiple GPUs Differ and Two 24GB GPUs vs One 48GB GPU for an LLM.