A100 vs H100 vs L40S, Explained for Beginners

Hardware guide · Educational content — all rates shown are illustrative examples, not offers

Why these three GPUs?

If you browse any GPU cloud's catalog, three names appear again and again: the NVIDIA A100, the NVIDIA H100, and the NVIDIA L40S. They represent three different generations and design priorities, and together they cover nearly every AI workload a developer is likely to run — from fine-tuning a small language model to serving inference at scale. Understanding what each one was built for is the fastest way to stop guessing and start choosing deliberately.

Two other GPUs appear in our cost calculator and are worth knowing: the V100, an older generation still useful for budget experimentation, and the RTX 4090, a consumer card that some clouds offer for cost-sensitive work. This guide focuses on the three data-center flagships, but the same thinking applies to all of them.

Head-to-head specifications

Here are the headline specs that matter most for AI work. These are the manufacturers' published figures:

Bandwidth figures are approximate published specs. "Typical strength" is a generalization — each GPU can do all of these tasks; the question is efficiency and cost.

The A100: the proven workhorse

Released on the Ampere architecture, the A100 has been the default data-center GPU for AI since 2020, and it remains enormously popular. It comes in 40 GB and 80 GB versions; the 80 GB variant is the one you want for language models, because model weights and optimizer states eat memory fast.

The A100's strengths are maturity and versatility. Nearly every deep-learning framework, library, and tutorial targets it. Multi-Instance GPU (MIG) lets you split one A100 into up to seven isolated slices — handy for sharing a card across small experiments or users. For fine-tuning models in the 7B–70B parameter range, the A100 80GB is often the sweet spot: enough memory for serious work, widely available, and typically cheaper per hour than the H100 in illustrative pricing comparisons.

Its weakness is simply age: per watt and per dollar of hardware, newer cards move more data and compute faster. But "older" does not mean "obsolete" — for fine-tuning, inference of medium models, and most research experimentation, the A100 remains a thoroughly sensible choice.

The H100: the flagship for large-scale training

The H100, built on the Hopper architecture, is NVIDIA's flagship AI training GPU. Its headline upgrades over the A100 are HBM3 memory at around 3 TB/s of bandwidth and the Transformer Engine, which automatically mixes FP8 and FP16 precision to accelerate transformer models — the architecture behind virtually all modern language models.

In practice, the H100 shines brightest on large-scale training: pre-training or full fine-tuning of large language models across many GPUs. For those workloads it can substantially outperform the A100 per GPU, which is why it commands the highest hourly rates and is frequently the hardest GPU to get capacity for.

For smaller jobs, the H100 is often overkill. Fine-tuning a 7B model with LoRA — a task that fits comfortably on a single A100 or even smaller cards — gains little from an H100's extra muscle while paying the flagship premium. A useful rule of thumb: reach for the H100 when your workload is limited by training time on many GPUs, not when a single cheaper GPU would finish the job overnight anyway.

The L40S: the inference and graphics specialist

The L40S is built on the Ada Lovelace architecture — the same generation as the RTX 40-series consumer cards — but packaged as a data-center PCIe card with 48 GB of GDDR6 memory. It is the most versatile of the three in one specific sense: it handles AI inference, graphics rendering, and video workloads well, making it popular for applications that mix them.

For inference, the L40S is frequently the value pick. Inference is usually memory-capacity-bound (can the model fit?) rather than compute-bound, and 48 GB fits a wide range of open models in quantized form. Its fourth-generation Tensor Cores and FP8 support keep throughput strong, while its illustrative hourly rates typically sit well below the H100's.

The trade-off is memory bandwidth: at roughly 864 GB/s, it moves data considerably slower than the HBM-equipped A100 and H100. For memory-bandwidth-hungry training of large models, that matters. For serving inference or fine-tuning modest models, it often does not — which is exactly why the L40S has become a favorite for production inference deployments.

Why VRAM matters more than you think

Beginners often compare GPUs by raw compute speed. Experienced practitioners compare them by VRAM capacity first. Here is why: if your model does not fit in GPU memory, you cannot run it at all on that card — no amount of speed helps. A useful approximation for inference is that each billion parameters needs roughly 2 GB of VRAM in 16-bit precision (so a 70B model needs on the order of 140 GB, requiring multiple GPUs or quantization), plus extra headroom for the KV cache and activations.

Training needs far more memory than inference — commonly 3–4× the model's weight size once you count gradients and optimizer states — which is why full fine-tuning of large models demands 80 GB cards or multi-GPU setups, while parameter-efficient methods like LoRA slash the requirement dramatically. Our guide to estimating GPU hours walks through this math in detail.

The practical takeaway: choose the cheapest GPU whose VRAM fits your workload, not the fastest GPU you can afford. An L40S that fits your model beats an H100 you cannot fully utilize, and a 40 GB A100 that fits beats an 80 GB card you do not need.

Which one should you pick?

Walk through these questions in order:

1. What is the workload? Training from scratch → lean H100 (at scale) or A100. Fine-tuning → A100 80GB or L40S, or smaller cards with LoRA. Inference → L40S or whatever fits the model cheapest.
2. Does the model fit in VRAM? Estimate memory needs first. If not, your options are quantization, a bigger card, or multiple GPUs (see our multi-GPU guide).
3. What is the budget per run? Plug candidate GPUs into the cost calculator with illustrative rates. A slower, cheaper GPU that finishes overnight can beat a flagship on cost per completed job.
4. Is capacity available? The "best" GPU is the one you can actually rent. Have a fallback GPU in mind — flexibility across two or three GPU types is a superpower when flagship capacity sells out.

And one final piece of advice: benchmark your own workload. Published specs and rules of thumb get you to a shortlist, but a one-hour test run on each candidate GPU — measuring tokens per second, time per epoch, or whatever your metric is — gives you ground truth for your specific code, model, and data. An hour of testing routinely saves hundreds of dollars of mis-sized rental.

Related guides

• How GPU Cloud Pricing Works: On-Demand vs Spot vs Reserved
• How to Estimate GPU Hours for Training and Fine-Tuning
• Glossary of GPU Terms