In this article
QLoRA combines adapter training with a quantized base, but weight storage alone does not determine training memory.
LoRA and QLoRA differ in how the base model is represented: LoRA trains added adapters while the base is frozen, and QLoRA combines that approach with quantized base weights. GPU memory also includes activations, gradients, optimizer state and working buffers. A weight-only estimate is therefore insufficient for a training requirement.
Inventory memory and trainable parts
Record the base model and revision, target modules, adapter rank and the parameters that are actually trainable. PEFT’s LoRA documentation explains the adapter approach, but the method name does not define an identical allocation for every architecture or configuration.
Inspect the configured model rather than assuming every parameter outside an adapter is frozen under every setup. Preserve the training recipe, input-length distribution and numeric formats. These facts allow another operator to interpret a memory measurement and reproduce the experiment.
Compare methods by their structure
| Approach | What is trained | Base state | Separate checks |
|---|---|---|---|
| Full fine-tuning | Main model parameters | Chosen training format | Base gradients and optimizer state |
| LoRA | Selected adapters and permitted modules | Usually frozen | Rank, activations and target layers |
| QLoRA | Adapters over a quantized base | Quantized and frozen | Backend support and compute precision |
The table describes approaches, not a universal performance ranking. A published capacity example cannot be transferred automatically to another architecture, sequence length or batch. Preserve those conditions when comparing with a reference experiment.
Treat quantization as configuration
PEFT documents adapter training over a quantized model. Storage representation and computation format should be recorded separately. Four-bit weights do not imply that every datum in the training process occupies half a byte per parameter; activations and other state have their own requirements.
Verify support for the selected backend and recipe. Compare the same task and held-out evaluation when changing quantization. Lower memory is useful only if the trained adapter remains suitable for the intended purpose. Do not infer that a model which fits for inference necessarily supports the chosen training step.
Measure the complete step and outcome
Run a short controlled sequence including forward computation, backward computation and optimizer updates. Observe later steps as well as the first, and include checkpoint saving and evaluation where relevant. Keep peak VRAM, RAM, time and errors with the exact configuration.
Evaluate the result against the unadapted baseline on held-out data and test reloading the adapter. A training method does not itself establish useful improvement, demand or financial return.
Official references: huggingface.co — lora, huggingface.co — quantization, arxiv.org — 2305.14314.
Related guides: Benchmarking GPUs for Stable Diffusion reproducibly, A bounded PEFT and LoRA experiment on your GPU.