Multi-GPU Setups: NVLink, Scaling, and When You Need More Than One GPU
On this page
Why use more than one GPU?
There are exactly two reasons to rent multiple GPUs instead of one: your model does not fit on a single GPU, or one GPU is too slow for your deadline. Everything else is detail. If neither is true — the model fits and finishes in time — a single GPU is simpler, cheaper, and easier to debug. Many developers reach for multi-GPU setups out of habit or ambition; the disciplined question is always whether one GPU would do.
That said, multi-GPU training is the backbone of modern AI. Large language models are trained on hundreds or thousands of GPUs working together, and even modest projects outgrow a single card. Understanding how the work gets split — and what it costs in efficiency — is essential knowledge once you cross that line.
The two ways to split work: data vs model parallelism
Data parallelism
In data parallelism, every GPU holds a complete copy of the model, and each processes a different slice of the data batch. After each step, the GPUs synchronize their gradients (usually via an all-reduce operation) so all copies stay identical. It is the simplest multi-GPU strategy and the default in most frameworks' distributed training.
Data parallelism works best when the model fits comfortably on one GPU and you want to go faster by processing more data at once. Its main cost is communication: gradients must travel between GPUs every step, and that transfer time grows with model size.
Model (tensor) parallelism
In model parallelism (often called tensor parallelism), the model itself is split across GPUs — each card holds part of the layers or part of each layer's weights. This is how models larger than a single GPU's VRAM get trained and served. The trade-off is much heavier communication: GPUs must exchange activations constantly, not just gradients once per step.
In practice, large training runs combine both: tensor parallelism within a node (where the interconnect is fast) and data parallelism across nodes. Frameworks like DeepSpeed and Megatron-LM automate these strategies, but understanding what they are doing helps enormously when something is slow.
The interconnect: NVLink vs PCIe
GPUs in the same machine talk to each other over an interconnect, and its speed shapes everything about multi-GPU performance:
- NVLink is NVIDIA's high-bandwidth GPU-to-GPU interconnect, with per-link bandwidth in the hundreds of GB/s (bidirectional figures vary by generation). On SXM-based systems like H100 and A100 servers, NVLink (often with NVSwitch connecting many GPUs) lets 8 GPUs behave almost like one giant GPU for communication-heavy work.
- PCIe is the standard expansion-bus interconnect. Modern PCIe generations offer tens of GB/s — roughly an order of magnitude slower than NVLink for GPU-to-GPU traffic. PCIe-based multi-GPU systems (common with L40S and RTX cards) work fine for data parallelism but struggle with heavy tensor parallelism.
The practical consequence: match the interconnect to the parallelism strategy. Tensor-parallel training of large models wants NVLink-class bandwidth; data-parallel fine-tuning is usually fine on PCIe. When a cloud lists an "8× GPU" machine, check whether it is SXM/NVLink or PCIe — the same 8 GPUs can behave very differently.
Across machines (multi-node), GPUs communicate over the network — typically InfiniBand on high-end clusters or Ethernet elsewhere. Multi-node training adds meaningful complexity and is generally only worthwhile for large-scale runs; most developers should exhaust single-node options first.
Scaling efficiency and diminishing returns
Scaling efficiency measures how much faster N GPUs are compared to one. Perfect scaling would be N×; reality is always less, because of communication overhead, synchronization stalls, and parts of the workload that cannot be parallelized.
As an illustrative intuition — not a rule — data-parallel training on well-connected GPUs might achieve on the order of 80–90% efficiency going from 1 to 2–4 GPUs, with efficiency declining as you add more. Doubling from 4 to 8 GPUs might yield only 1.5–1.7× more throughput in a communication-heavy workload. The exact numbers depend on model size, batch size, interconnect, and framework tuning; the pattern of diminishing returns is universal.
This matters for cost because you pay per GPU-hour regardless of efficiency. Eight GPUs at 70% scaling efficiency cost 8× the hourly rate while delivering ~5.6× the throughput of one — a worse cost-per-unit-of-work than fewer GPUs. The economically optimal GPU count is often smaller than the fastest-feeling one. When budgeting multi-GPU runs, discount your throughput expectations accordingly (see our estimation guide).
When a single GPU is enough
More often than beginners expect. A single GPU suffices when:
- The model and a reasonable batch fit in VRAM (use quantization or LoRA if close).
- The job finishes within your deadline — overnight runs are fine for most experimentation.
- You are still iterating on code and data. Multi-GPU debugging multiplies confusion; get it working on one GPU first.
- You are doing inference for a prototype or low-traffic service. One GPU serving a quantized model handles a surprising amount of traffic.
A good rule: start on one GPU, and only scale out when you can name the constraint — "it doesn't fit" or "it's too slow for the deadline." Vague ambition is not a constraint.
The cost math of going multi-GPU
Multi-GPU cost is straightforward multiplication with one subtlety. The formula:
Cost ≈ hourly rate × number of GPUs × wall-clock hours — where wall-clock hours already reflect your (imperfect) scaling efficiency.
Example with illustrative numbers: suppose a job takes 40 hours on 1 GPU at an example rate of $2.00/hr → $80. On 4 GPUs with 80% scaling efficiency, wall-clock time is 40 ÷ (4 × 0.8) = 12.5 hours, and cost is $2.00 × 4 × 12.5 = $100. You finished 3× faster but paid 25% more. Whether that trade is worth it depends on how much you value your time — for deadline-driven work, often yes; for background experimentation, often no.
Try these trade-offs yourself in the GPU cost calculator: enter different GPU quantities and compare the illustrative totals.
Practical tips for multi-GPU rentals
- Prefer full nodes for tensor parallelism. An 8-GPU node with NVLink gives you the fast interconnect; cobbling together GPUs across machines does not.
- Verify GPU topology. On any multi-GPU machine, check how the GPUs are connected (framework tools can report this) before assuming NVLink speeds.
- Scale batch size with GPU count in data parallelism to keep per-GPU work constant; tiny per-GPU batches waste the extra cards.
- Checkpoint aggressively. Multi-GPU runs have more failure modes (one bad GPU can kill the job). Frequent checkpoints to persistent storage are cheap insurance — and essential if you are using spot instances (see our pricing guide).
- Watch for stragglers. If one GPU is slower (thermal throttling, a bad card), synchronous training waits for it every step. Monitoring per-GPU utilization reveals this quickly.
- Consider whether fewer, bigger GPUs beat more, smaller ones. Two 80 GB cards with tensor parallelism sometimes outperform four 40 GB cards at similar cost — it depends on the workload, so test.