Choosing a GPU for Fine-Tuning: Memory Math for Full, LoRA and QLoRA Runs
Fine-tuning hardware is a memory question, and memory is a method question. Full fine-tuning needs several times the model's own size; LoRA and QLoRA need far less. This guide gives the arithmetic for each method, a sizing table by model size, and a pre-rental check.
All figures in this guide are illustrative examples for education. Real GPU prices, cloud rates and throughput vary by provider, region, model and time. Confirm current numbers with a provider before budgeting. The point here is the method; plug in your own numbers.
On this page
The four things that use memory
Inference needs memory for the weights and for the working state of the current request. Fine-tuning needs more, because it has to remember how to change the weights. The components:
- Weights. The model itself. At 16-bit precision, roughly 2 bytes per parameter.
- Gradients. One value per trainable parameter, recording how the loss changes with that parameter. Same size as the trainable weights.
- Optimiser state. Most common optimisers keep two extra values per trainable parameter, often at 32-bit precision, which can be the largest single component in full fine-tuning.
- Activations. The intermediate values from the forward pass that are needed during the backward pass. These scale with batch size and sequence length, and can be traded for compute with activation checkpointing.
The key phrase is trainable parameter. Methods that train only a small fraction of the parameters shrink the gradient and optimiser components by the same fraction.
How each method changes the sum
| Method | Weights | Gradients + optimiser | Rough total for a 7B model (illustrative) |
|---|---|---|---|
| Full fine-tune, 16-bit weights, 32-bit optimiser | ~14 GB | ~14 GB gradients + ~56 GB optimiser | 80 GB+ before activations; needs a large card or several |
| Full fine-tune with memory-saving optimiser | ~14 GB | Reduced, varies by optimiser | Often 40–60 GB |
| LoRA (adapters on 16-bit base) | ~14 GB | Small; adapters are a tiny fraction of parameters | ~18–24 GB with modest batch and sequence |
| QLoRA (adapters on 4-bit base) | ~4 GB | Small | ~8–12 GB; fits many consumer cards |
These totals are illustrative and generous with activations; your real number depends on sequence length, batch size and whether activation checkpointing is on. The pattern is reliable: full fine-tuning multiplies the model size several times over, LoRA roughly adds a margin to the inference footprint, and QLoRA shrinks the base model as well.
Sizing table by model size
| Model size | QLoRA | LoRA | Full fine-tune |
|---|---|---|---|
| 1–3B | 8 GB card | 12–16 GB card | 24–40 GB card |
| 7–8B | 12–16 GB card | 24 GB card | 80 GB card or multi-GPU |
| 13B | 16–24 GB card | 40 GB card | Multi-GPU |
| 30–34B | 24–40 GB card | 80 GB card | Multi-GPU cluster |
| 70B | 48–80 GB card | Multi-GPU | Multi-GPU cluster |
Illustrative; sequence length moves every cell. A run at 8,000-token sequences needs materially more activation memory than one at 1,000 tokens, and that can push a job one row down the table. When in doubt, estimate for the longest sequences in your data, not the average.
Choosing the method before the card
The method decides the card, so decide the method first. Questions that settle it:
- How different is your target task from what the model already does? Teaching a style, a format or domain vocabulary is usually well served by adapters. Changing fundamental behaviour across the board sometimes needs full fine-tuning.
- How much data do you have? Small datasets (thousands of examples) rarely justify full fine-tuning; adapters are less prone to over-fitting and far cheaper.
- Can you accept a small quality trade-off for a large cost cut? QLoRA’s 4-bit base can cost a little quality versus LoRA on a 16-bit base. Measure on your evaluation set, as described in quantization for cost planning.
- Will you serve the result? Adapters can be merged into the base for serving or kept separate and swapped. Full fine-tunes produce a new full model that has to be served in full.
For most teams fine-tuning for the first time, an adapter method on the largest base model that fits the available card is the right starting point, and full fine-tuning is the exception that needs a reason.
Verifying the estimate before you pay for the big card
- Rent the cheapest card that your estimate says should work, for one hour.
- Run a handful of training steps at your real sequence length and batch size, with the framework’s memory reporting on.
- If it fits with headroom, you have your card. If it runs out of memory, reduce batch size, enable activation checkpointing or shorten sequences, and try again before moving to a larger card.
- Note the step time; multiply by the number of steps for your full run to get the GPU-hour estimate. Our guide on estimating GPU hours covers the rest.
An hour on a small card costs little and answers the question that a day of forum reading cannot: does this model with this data fit on this GPU.
Common mistakes
- Estimating with the average sequence length and discovering the longest examples do not fit.
- Forgetting the optimiser state, which can be larger than the model in full fine-tuning.
- Renting a large card because the base model is large, when QLoRA would fit it on a card a quarter the price.
- Merging a QLoRA adapter into a 4-bit base for serving and being surprised by quality; merge into the full-precision base instead.
- Skipping the one-hour test and discovering memory errors on the expensive instance.