Inference vs Training: Why Serving a Model Usually Costs More Than Building It
Training is a one-time GPU bill; serving is a bill that runs for as long as users exist. This guide shows how to estimate both sides, why inference usually ends up the larger number, and which levers actually reduce serving cost.
All figures in this guide are illustrative examples for education. Real GPU prices, cloud rates and throughput vary by provider, region, model and time. Confirm current numbers with a provider before budgeting. The point here is the method; plug in your own numbers.
On this page
Two bills with very different shapes
Every AI product pays two GPU bills. The first is training (or fine-tuning): a burst of compute that runs for hours or days, finishes, and produces a set of weights. The second is inference: the compute that runs every time a user sends a request, for as long as the product exists. The first bill is a capital expense that you can plan, delay or skip by using an existing model. The second is an operating expense that scales with success.
Teams that come from a research background instinctively budget for the first bill because it is the dramatic one. Teams that have shipped a product know that the second bill is where the money goes. A fine-tune that costs a few hundred dollars (illustrative) can sit behind a service that costs that much every day once it has real traffic. This guide is about seeing both bills clearly before you commit to either.
Estimating the training bill
The training estimate has three inputs: how many GPU-hours the job needs, what one GPU-hour costs, and how many attempts you will make. Our guide on estimating GPU hours walks through the first input in detail. The short version is that GPU-hours scale with model size, dataset size and the number of passes over the data, and that efficient fine-tuning methods cut the number dramatically compared with training from scratch.
The third input is the one people forget. Nobody trains once. You will run a small pilot, discover a data problem, fix it, run again, adjust a hyperparameter, run again. A realistic training budget is the single-run estimate multiplied by three to five. If you plan for one run and do five, the training line is not over budget by a little; it is over by several hundred percent.
Illustrative example: a fine-tune estimated at 12 GPU-hours on a card you can rent for $2 per hour is $24 per run. Five runs is $120. Add storage for checkpoints and a few hours of a cheaper GPU for evaluation, and a sensible budget for this job is in the low hundreds of dollars, not the $24 the first calculation produced.
Estimating the inference bill
The inference estimate needs different inputs: how many requests you serve, how much compute each request consumes, and how efficiently your serving setup turns GPU time into finished requests. For language models the useful unit is tokens, because cost is roughly proportional to tokens processed, with output tokens costing more than input tokens because they are generated one at a time.
A simple serving model looks like this:
| Input | Illustrative value | Where it comes from |
|---|---|---|
| Requests per day | 50,000 | Your product analytics or forecast |
| Average tokens per request (in + out) | 1,200 | Measure on real traffic; guesses are usually low |
| Tokens per second one GPU sustains | 1,500 | Benchmark your model on your serving stack at realistic batch size |
| GPU hourly rate | $2.50 | Provider pricing page |
| Utilisation you actually achieve | 40% | Traffic is bursty; GPUs idle between peaks |
With those numbers: 50,000 × 1,200 = 60 million tokens per day. At 1,500 tokens per second a fully busy GPU handles 5.4 million tokens per hour, so the work needs about 11 GPU-hours per day of actual compute. At 40 percent utilisation you are paying for roughly 28 GPU-hours to get those 11, which is about $70 per day or around $2,100 per month. The $24 fine-tune is paid for before lunch on the first day.
Every number in that table is illustrative, but the structure is not. Note which inputs you control: tokens per request (prompt design), tokens per second (model size, quantization, serving software, batching) and utilisation (autoscaling, batching, scheduling). The hourly rate is the input people negotiate hardest over and it is often the smallest lever.
The ratio most teams discover too late
Divide the monthly inference bill by the one-off training bill and you get a ratio that tells you where to spend engineering effort. For a product with modest traffic the ratio might be close to one; for a product with real users it is commonly ten or more within the first year, because inference accumulates every month and training does not.
The practical consequence is counter-intuitive. It can be worth spending more on training (a longer fine-tune, a distillation step, a smaller model trained harder) if the result is a model that is cheaper to serve. A one-time cost that halves a recurring cost pays back quickly. Conversely, choosing a larger model because it was easy to fine-tune, then serving it for a year, is the most expensive decision in the whole pipeline and usually the least examined.
Levers that cut serving cost, in rough order of impact
- Use a smaller model where it is good enough. Serving cost scales with parameters. If a smaller model meets your quality bar on your evaluation set, the saving is proportional and permanent.
- Cut tokens per request. Trim system prompts, cap output length, stop sending the whole conversation history when the last few turns will do. This is free and often worth tens of percent.
- Batch requests. GPUs are most efficient when they process many requests at once. Serving software that batches continuously can multiply throughput compared with one-request-at-a-time serving.
- Quantize. Lower-precision weights use less memory and often run faster, at a quality cost you should measure rather than assume. See quantization for cost planning.
- Raise utilisation. Autoscale on a queue-length signal, schedule batch jobs into idle periods, and switch off replicas you do not need overnight. Paying for idle GPUs is the most common waste on inference bills.
- Negotiate the rate. Reserved capacity, committed-use discounts and cheaper regions help, but only after the levers above, because they lock in whatever inefficiency you still have.
Putting both bills in one plan
Write the two estimates side by side with the assumptions visible. For training: GPU-hours per run, cost per hour, number of runs, storage. For inference: requests, tokens per request, throughput, utilisation, hourly rate, replicas for redundancy. Then run three scenarios for inference, at a third of your forecast traffic, at the forecast, and at three times it. If the three-times scenario is unaffordable, that is the moment to pick a smaller model or a different architecture, before the fine-tune, not after.
Our GPU cost calculator handles the GPU-hour arithmetic for either bill; the forecasting inputs have to come from you. When in doubt, over-estimate tokens per request and under-estimate utilisation; those are the two inputs that are almost always worse in production than in the spreadsheet.