Quantization for Cost Planning: FP16 to 4-Bit, What Changes and What It Costs
Quantization is the single most effective way to fit a model on a cheaper GPU — or serve more requests from the same one. Here is the precision ladder in plain terms: what each step saves, what it risks, and how to judge the trade-off.
All figures in this guide are illustrative examples for education. Real GPU prices, cloud rates and throughput vary by provider, region, model and time. Confirm current numbers with a provider before budgeting. The point here is the method; plug in your own numbers.
On this page
What quantization actually does
A model’s weights are numbers. Storing each number with more bits gives more precision and takes more memory; storing it with fewer bits takes less memory and loses a little precision. Quantization is the process of converting weights (and sometimes the activations that flow through the model at run time) from a higher-precision format to a lower one.
The reason this matters for cost is memory. A model with seven billion parameters at 16 bits per parameter needs roughly 14 GB just for weights; at 8 bits it needs about 7 GB; at 4 bits about 3.5 GB, plus overhead for the model’s working memory. Those thresholds decide which GPU can hold the model at all, and the GPU decides the hourly rate. See how much VRAM you need for the sizing tables.
The precision ladder
| Format | Bits per weight | Memory vs FP16 | Typical use | Quality risk |
|---|---|---|---|---|
| FP32 | 32 | 2× | Training of small models, numerically sensitive work | None; this is the reference |
| FP16 / BF16 | 16 | 1× | Standard training and serving precision | Negligible for most models |
| FP8 | 8 | 0.5× | Serving and increasingly training on recent GPUs | Low, hardware-dependent |
| INT8 | 8 | 0.5× | Serving on a wide range of hardware | Low with good calibration |
| 4-bit (various schemes) | 4 | 0.25× | Fitting large models on small GPUs; local inference | Noticeable on some tasks; measure |
| 3-bit and below | ≤3 | ≤0.19× | Extreme memory constraints | Often significant |
The memory column is approximate because real quantization schemes keep some parts of the model at higher precision and add small amounts of metadata. The quality column is deliberately vague because it depends on the model, the task and the scheme; the next section is about replacing it with your own measurement.
Memory savings versus speed savings
Lower precision always saves memory. It does not always save time. Whether a quantized model runs faster depends on whether the GPU has hardware for that precision and whether the serving software uses it. On a card with native 8-bit matrix units, INT8 or FP8 serving can run substantially faster than FP16; on a card without them, the same model may run at the same speed or slower because weights are unpacked to higher precision before each operation.
For cost planning this means two separate questions. First: does quantization let me use a cheaper card? That is a memory question and the answer is usually yes. Second: does it let me serve more requests per hour on the card I have? That is a hardware-and-software question and the only reliable answer is a benchmark on your stack.
Measuring the quality cost instead of guessing
Claims like “4-bit loses almost nothing” are true for some models on some benchmarks and false for yours until proven otherwise. The method:
- Build an evaluation set of a few hundred examples that look like your real traffic, with answers or scoring rules you trust.
- Run the FP16 model and record the score. This is your baseline.
- Run each quantized version you are considering on the same set and record the score and the throughput.
- Decide the largest quality drop you can accept before you look at the cost savings, or the savings will decide for you.
- Pick the cheapest configuration that stays inside that limit.
Pay attention to the tails, not just the average. Quantization often leaves the typical case alone and damages specific capabilities: long-context reasoning, numerical answers, less common languages. If your product depends on one of those, test it explicitly.
A worked cost example
Illustrative: a 13-billion-parameter model at FP16 needs around 26 GB for weights plus working memory, which pushes it onto a 40 GB or 80 GB card renting at, say, $2.50 per hour. At 8-bit the same model needs about 13 GB plus overhead and fits on a 24 GB card renting at, say, $0.80 per hour. If throughput per card is similar, the hourly cost drops by two-thirds; if the smaller card is slower per request, the saving shrinks and you need the benchmark to know by how much.
At 4-bit the model fits on a 12 GB or 16 GB card at an even lower rate, but this is where quality measurement matters most. A product that answers customer questions may accept the trade; one that produces figures for an invoice may not.
When not to quantize
- When the model already fits comfortably on the card you need for other reasons and throughput is not the bottleneck.
- When your evaluation shows a quality drop that matters and you cannot recover it with a better scheme or calibration.
- During training and fine-tuning of the main weights, unless you are deliberately using a method designed for it; precision matters more when gradients are involved.
- When the engineering time to validate it exceeds the saving for a short-lived project.
Quantization is a tool for cost, and like any cost tool it should be applied after you have measured, not before.