GPU Utilisation: Why Your Bill Is Higher Than Your Compute
You pay for GPU-hours; your models use only some of them — and that gap is usually the largest line on an AI bill. This guide shows how to measure utilisation, where it leaks, and what a realistic target looks like for training, batch jobs, and serving.
All figures in this guide are illustrative examples for education. Real GPU prices, cloud rates and throughput vary by provider, region, model and time. Confirm current numbers with a provider before budgeting. The point here is the method; plug in your own numbers.
On this page
The gap you are paying for
A GPU instance bills by the hour it exists, not by the hour it computes. If a training job spends a third of its time loading data, a fifth of its time writing checkpoints and a few hours waiting for someone to notice it finished, you paid for all of that at the full rate. On inference, a replica that sits ready for a peak that lasts two hours a day is idle for twenty-two.
Utilisation is the ratio of useful compute to paid time. Across the teams whose set-ups we hear about, the honest number is often well below half, and the people running those jobs usually believe it is much higher because they look at the wrong gauge. Raising utilisation from 35 percent to 70 percent halves the bill with no change to the hourly rate, which is why it belongs above provider shopping on any cost plan.
Measuring it properly
The number that nvidia-smi shows as “GPU-Util” is a coarse signal: it reports whether a kernel was running during the sample window, not how much of the chip was busy. A job can show 100 percent on that gauge while using a fraction of the available compute because its kernels are small or memory-bound.
Better signals, from easiest to most precise:
- Wall-clock accounting. Log when the instance started, when useful work started, when it ended, and when the instance was stopped. The ratio of the middle interval to the whole is your coarse utilisation, and it is often the most shocking number.
- Throughput over time. Tokens per second, samples per second or requests per second, sampled every minute and plotted. Flat stretches at zero are idle time; stretches well below peak are bottlenecks.
- Profiler traces. Framework profilers show how much of each step is GPU compute versus data loading, host-to-device copies, synchronisation and Python overhead. Use them once per job type; they are expensive to run continuously.
- Power draw and SM activity. Sustained power near the card’s rating and high streaming-multiprocessor activity indicate the chip is actually working, as opposed to merely busy.
The six places utilisation leaks
| Leak | Typical symptom | Usual fix |
|---|---|---|
| Data loading | GPU waits for the next batch | More loader workers, pre-processed datasets, faster storage, prefetching |
| Checkpoint writes | Periodic stalls; long tail at job end | Asynchronous saves, less frequent checkpoints, faster disk |
| Small batches | Kernels finish before the chip is full | Larger batch sizes, gradient accumulation, continuous batching for serving |
| Human latency | Finished job sits idle until someone acts | Auto-shutdown on completion, job queues, alerts |
| Over-provisioned serving | Replicas ready for a peak that rarely comes | Autoscaling on queue length, scheduled scale-down, batch work in off-peak hours |
| Development on expensive cards | Interactive sessions on training-class GPUs | Develop on a small card or CPU; promote to the big card only for runs |
Human latency deserves a special mention because it is invisible in any profiler. A job that finishes at 2 a.m. and is terminated at 9 a.m. has wasted seven GPU-hours, and it happens every night in teams that do not automate shutdown.
Realistic targets by workload
Perfect utilisation is not achievable and chasing it has its own cost. Illustrative ranges that experienced teams treat as healthy:
- Long training runs: 70 to 85 percent of wall-clock time doing compute, once the data pipeline is sorted. Below 50 percent, profile before you buy more GPUs.
- Batch inference and offline jobs: 80 percent or better, because the work can be queued and packed. There is little excuse for idle GPUs on work that is not latency-sensitive.
- Online serving with bursty traffic: 30 to 60 percent, depending on how much latency headroom you need. Pushing higher means users wait at peaks; the trade-off is a product decision, not an engineering one.
- Interactive development: very low, by nature. The fix is not to raise it but to move development to cheaper hardware.
A worked example
Illustrative: a team runs a nightly fine-tune that takes six hours of compute on an instance billed at $3 per hour. The instance is started by hand at 6 p.m., the job starts at 6:30 after environment set-up, finishes at 12:30 a.m., and the instance is stopped at 9 a.m. the next morning. Paid: 15 hours, $45. Useful: 6 hours. Utilisation: 40 percent.
Automating start, set-up and shutdown brings paid time to about 6.5 hours, $19.50, before touching the job itself. Fixing a data-loader bottleneck that the profiler shows is costing 25 percent of each step brings the job to 4.5 hours. Paid: about $15 per night, a third of the original bill, on the same hardware at the same rate.
Habits that keep it high
- Every job logs its own start, first useful step, last useful step and end, and the ratio is reviewed weekly.
- No GPU instance exists without an owner and an auto-shutdown rule.
- Profile each new job type once, fix the largest stall, move on; do not tune forever.
- Serving autoscales on a signal that reflects actual demand (queue depth or latency), not on a schedule somebody set in March.
- Batch work has a queue, and the queue drains in off-peak hours on the same replicas that serve users by day.
These are organisational habits more than technical ones. They are also the difference between a bill that tracks your compute and a bill that tracks your calendar.