Batch vs Real-Time Inference: Cost per 1,000 Requests, Explained
Same model, same GPU, very different cost per request. Real-time inference answers while a user waits; batch inference processes a queue when convenient. Here is how to compute cost per thousand requests for each — and how to route most work the cheap way.
All figures in this guide are illustrative examples for education. Real GPU prices, cloud rates and throughput vary by provider, region, model and time. Confirm current numbers with a provider before budgeting. The point here is the method; plug in your own numbers.
On this page
Two modes, one model
Real-time (online) inference means a request arrives and a response must come back within a latency budget, typically well under a few seconds for an interactive product. Batch (offline) inference means requests accumulate in a queue and are processed in bulk whenever capacity is available: overnight classification of yesterday’s tickets, embedding a document archive, generating descriptions for a product catalogue.
The model, the GPU and the output are identical. What differs is how efficiently the GPU can be used. Real-time serving must keep capacity ready for whatever arrives, and must answer small groups of requests quickly rather than large groups efficiently. Batch serving can fill the GPU completely, run at the quietest hour on the cheapest instance, and tolerate an interruption. That difference shows up directly in cost per request.
Cost per thousand requests
For either mode the arithmetic is the same:
Cost per 1,000 requests = (GPU hourly rate ÷ requests completed per GPU-hour) × 1,000
The number that differs is requests per GPU-hour, which depends on batch size, utilisation and the instance type you can get away with. Illustrative comparison for one model:
| Real-time | Batch | |
|---|---|---|
| Instance type | On-demand, $2.50/h | Spot, $0.90/h |
| Effective batch size | Small (latency-bound) | Large (throughput-bound) |
| Requests per GPU-hour at full load | 9,000 | 30,000 |
| Utilisation achieved | 40% | 85% |
| Requests actually completed per paid hour | 3,600 | 25,500 |
| Cost per 1,000 requests | ~$0.69 | ~$0.035 |
In this illustration batch is roughly twenty times cheaper per request. The exact multiple will differ for your model and traffic, but a gap of several times is normal, and it comes from three stacked effects: a cheaper instance, larger batches, and higher utilisation.
Which requests actually need real time?
Fewer than most teams assume. The test is whether a person is waiting for the answer right now. A chat reply, an autocomplete suggestion and a search result are real time. A summary that appears in tomorrow’s digest, a moderation decision on content that is held for review anyway, embeddings for a nightly index rebuild, and classification of last week’s support tickets are not.
Go through your request types and mark each as “someone is waiting” or “nobody is waiting”. Products often find that half or more of their total volume falls in the second group and is being served through the expensive path out of habit.
Designing the split
- Give every request a deadline. Real-time requests get a latency budget; batch requests get a completion window (within the hour, by morning, by Monday).
- Route by deadline. Requests with a short budget go to the online replicas. Everything else goes to a queue.
- Drain the queue on the cheapest capacity. Spot instances, off-peak hours, large batches. Let the queue grow during the day and empty at night if the deadlines allow.
- Let batch overflow onto idle online capacity. When the online replicas are quiet, pull work from the queue. This raises their utilisation and shrinks the batch fleet.
- Keep the two paths on the same model version. Divergence between online and batch behaviour is a support problem waiting to happen.
A worked example
Illustrative: a product handles 200,000 requests per day. All of them currently go through online serving at about $0.69 per thousand, which is roughly $138 per day. Analysis shows 120,000 of those are notifications and digests that nobody waits for. Moving them to batch at about $0.035 per thousand costs about $4.20 per day; the remaining 80,000 online requests cost about $55. Total: around $59 per day instead of $138, with no change to the model or the user-facing latency.
The saving came entirely from classifying requests honestly. The engineering was a queue and a scheduler, not a new model.
Caveats
- Batch processing introduces delay by design. If a “nobody is waiting” request later turns out to have someone waiting, you will hear about it. Give batch requests a way to be escalated.
- Very large batches can hit memory limits and increase per-request latency inside the batch; the throughput gain flattens past a point you should find by measurement.
- Spot interruptions mid-batch need the same resumability as training jobs. See spot instances and checkpointing.
- Cost per thousand requests should be recomputed when the model, the prompt length or the traffic mix changes; the figure goes stale quickly.