Choosing a GPU for Inference Serving: Latency, Throughput and Cost per Request

Planning guide · Last updated: October 18, 2026 · Reviewed by the editorial team · All figures shown are illustrative examples, not offers

Training GPUs are chosen by memory and time-to-finish; serving GPUs are chosen by memory, throughput, and latency — and the cheapest card that meets all three is usually not the fastest one. This guide gives a method for sizing a serving deployment and choosing its hardware.

Three numbers decide the card

Memory. The model’s weights plus the working memory for the requests being processed at once must fit on the card (or across the cards). Working memory grows with batch size and context length, so a model that fits for one short request may not fit for thirty-two long ones. Our VRAM sizing guide covers the arithmetic.

Throughput. Tokens generated per second across all concurrent requests. Higher throughput per GPU means fewer GPUs for the same traffic and a lower cost per request. Throughput depends on the card, the model size and precision, the serving software and the batch size.

Latency. How long one user waits: time to first token and time per subsequent token. Latency is traded against throughput: larger batches raise throughput and raise per-request latency. The product decides the acceptable latency; engineering works backwards from it.

A card that wins on throughput can lose on cost if a cheaper card also meets the latency target. The right question is not “which GPU is fastest” but “which GPU meets the latency target at the lowest cost per thousand requests”.

A sizing method in six steps

1. Fix the latency target. For example: first token within 800 ms, then at least 20 tokens per second per user, at the 95th percentile. Write it down; everything else follows from it.
2. Measure traffic. Requests per second at peak, average and maximum input and output tokens. Guesses here are usually low; measure on real traffic or a realistic simulation.
3. Pick candidate cards. Every card with enough memory for the model at your chosen precision, from the cheapest up. Include the cards one quantization step down; see quantization for cost planning.
4. Benchmark each candidate on your serving stack. Increase concurrent requests until the latency target is breached. The throughput just before that point is the card’s usable capacity for your workload.
5. Compute cost per thousand requests = (hourly rate ÷ usable requests per hour) × 1,000, for each card. Our batch versus real-time guide shows the arithmetic.
6. Size the fleet. Peak requests per second ÷ usable capacity per card, rounded up, plus one for redundancy. Then check whether autoscaling can shrink it off-peak.

An illustrative comparison

The numbers are invented to show the shape of the result, which is common in practice: a mid-range card at reduced precision wins on cost, the largest card wins on simplicity, and the smallest card loses because it cannot hold enough concurrent requests before latency breaks. Your table will have different numbers; build it rather than assuming it.

Trade-offs beyond the table

• Fleet size versus card size. Ten small cards give finer autoscaling and better fault tolerance than two large ones, at the cost of more orchestration. For bursty traffic, finer scaling often saves more than raw efficiency.
• Memory headroom. A card that fits the model with 5 percent to spare will fall over on the first long request. Leave room for the longest context you allow, not the average.
• Quality at reduced precision. Measure it on your evaluation set before you commit to the cheaper card. A cost saving that comes with a quality complaint is not a saving.
• Availability. The cheapest card is sometimes the one in shortest supply. A serving fleet needs cards you can actually get at 3 a.m. when a replica fails. See regions and availability.
• Multi-model serving. If several models share a fleet, size for the largest and check that the rest can co-reside without starving each other of memory.

Common mistakes

• Choosing the training card for serving because it is already rented. Serving runs for months; the wrong card costs more every day.
• Benchmarking with one request at a time. Serving efficiency comes from concurrency; single-request numbers are irrelevant to cost.
• Sizing for average traffic. Users arrive at the peak; the fleet must cope with it or the latency target is fiction.
• Ignoring utilisation. A fleet sized for peak and left running all night is paying for idle capacity; see utilisation.
• Skipping redundancy. One replica is a plan to be down.

A checklist before you commit

• Latency target written down with a percentile, not an average.
• Traffic measured at peak with token counts, not estimated.
• At least three candidate cards benchmarked on the real serving stack at realistic concurrency.
• Cost per thousand requests computed for each, with utilisation included.
• Quality at the chosen precision verified on your own evaluation set.
• Fleet sized for peak plus redundancy, with an autoscaling plan for off-peak.

The GPU cost calculator handles the GPU-hour arithmetic once you have the usable capacity figure from your benchmark.

Related guides

• Inference vs Training: Why Serving a Model Usually Costs More Than Building It
• Batch vs Real-Time Inference: Cost per 1,000 Requests
• Quantization for Cost Planning: FP16 to 4-Bit, What Changes
• A100 vs H100 vs L40S, Explained for Beginners