AI Cluster Interconnect Bandwidth Estimator
Interactive planning aid · Runs entirely in your browser — nothing you enter leaves your device
AI cluster interconnect bandwidth is the rate at which the GPUs in a cluster can exchange data with each other — gradients, activations and model shards — and it is the number that decides whether a cluster spends its time computing or waiting. This estimator turns your GPU type, cluster size and model size into a per-GPU scale-out bandwidth figure, a recommended network adapter tier, node and rack counts, and a plain-English verdict on where copper stops working and optical interconnects become necessary.
Educational estimates only. Every figure on this page is an order-of-magnitude planning estimate built from public vendor specifications and the stated assumptions in How this estimator works. It is not an engineering design, a quote, or a substitute for validation by your infrastructure provider.
Estimated scale-out bandwidth needed per GPU
AI cluster interconnect bandwidth, explained
AI cluster interconnect bandwidth is the rate, in gigabits or gigabytes per second, at which the GPUs in a cluster can exchange data with one another. It comes in two tiers that differ by more than an order of magnitude. Scale-up bandwidth connects GPUs inside one tightly coupled domain — a single node, or a rack-scale domain such as the 72-GPU NVLink domain of a GB200 NVL72 — using NVLink or AMD Infinity Fabric at 900 GB/s to 1,800 GB/s per GPU over backplanes and short copper cables. Scale-out bandwidth connects nodes to each other through InfiniBand or Ethernet network adapters at 200G, 400G or 800G per GPU, almost always over optical links, because passive copper cables can only carry these lane speeds for about 2 to 3 meters. Training a large language model stresses both tiers: tensor parallelism rides the scale-up fabric, while data-parallel gradient synchronization, pipeline stages and checkpoint traffic ride the scale-out network. If scale-out bandwidth is undersized, expensive GPUs idle while they wait for gradients, which is why interconnect planning starts from the training math in the methodology below rather than from a catalog.
Key facts
- An NVIDIA H100 SXM provides 900 GB/s of NVLink scale-up bandwidth per GPU; a B200 provides 1,800 GB/s. AMD's MI300X and MI325X provide 896 GB/s of Infinity Fabric bandwidth per GPU inside an 8-GPU mesh.
- The GB200 NVL72 extends the scale-up domain to 72 GPUs in one rack, all connected at full NVLink bandwidth; most other platforms have a scale-up domain of 8 GPUs (one node).
- Data-parallel training moves about 4 bytes of gradient data per parameter per GPU per step (ring all-reduce factor of roughly 2, times 2 bytes per BF16 gradient).
- At 800G lane speeds, passive direct-attach copper (DAC) reaches roughly 2 to 3 meters; anything beyond the rack is carried by optical links.
- An 800G pluggable optical module draws roughly 14 to 18 W; two modules per link means roughly 28 W per link at an energy cost of about 15 to 20 picojoules per bit.
- Co-packaged optics (CPO) targets roughly 5 to 10 picojoules per bit — about half the energy per bit of pluggable modules — by moving the optical engine next to the switch or GPU package.
- Scale-out network adapters for AI clusters ship in 200G, 400G and 800G generations (ConnectX-7, ConnectX-8 / SuperNIC class); 1.6T is the next step on vendor roadmaps.
How this estimator works
For training and full-parameter fine-tuning, the estimator models the traffic that is easiest to derive from first principles: data-parallel gradient synchronization. It works in four steps, and every assumption is visible so you can challenge any of them.
- Gradient bytes per step. In data-parallel training, every GPU computes gradients for the full model on its own slice of data, then all GPUs average their gradients with an all-reduce before the next step. A ring all-reduce moves about twice the data it reduces, and BF16 gradients occupy 2 bytes per parameter, so the bytes each GPU must exchange per step are approximately 2 × parameters × 2 bytes. For a 70B model that is about 280 GB per GPU per step.
- Steps per second. Training a transformer costs roughly 6 FLOPs per parameter per token (two for the forward pass, four for the backward pass). A GPU sustaining a fraction of its peak dense BF16 throughput — the model FLOPs utilization, or MFU — therefore processes (peak TFLOPS × MFU) ÷ (6 × parameters) tokens per second. Dividing by the batch tokens each GPU handles per step gives steps per second.
- Required scale-out bandwidth. Multiplying gradient bytes per step by steps per second gives the average cross-node bandwidth each GPU needs, in bytes per second; the tool reports it in gigabits per second and rounds it up to the nearest standard adapter tier (200G, 400G, 800G). One honest consequence of this algebra: the parameter count appears in both the traffic and the compute terms and cancels out, so the data-parallel estimate depends on the GPU, the MFU and the batch size — not on model size. Larger models still need more network in practice because they require more tensor and pipeline parallelism, which this estimator deliberately does not model.
- Fabric totals, topology and links. Total fabric bandwidth is the per-GPU figure multiplied by the GPU count (aggregate one-way egress). Node count assumes 8 GPUs per node; rack count assumes 4 nodes (32 GPUs) per rack for node-based platforms, or one 72-GPU rack for the GB200 NVL72. The optical link estimate divides total fabric bandwidth into 800G links, and the power comparison prices each such link three ways using published per-link figures: passive DAC copper at effectively 0 W per link, pluggable optics at about 28 W per link (two 800G modules at roughly 14 W each), and CPO at about 12 W per link (800 Gb/s at roughly 7.5 picojoules per bit). All three are labeled estimates, not specifications.
For inference, there is no gradient synchronization to model, so the estimator switches to a memory-fit rule of thumb. Model weights in BF16 occupy about 2 bytes per parameter; the tool divides the weight size by the HBM capacity of one GPU to get the minimum GPU count that can hold the model, rounds up to whole nodes of 8 GPUs, and compares that with the GPU count you entered. The scale-out recommendation for serving is a rule of thumb — 200G per GPU when the deployment fits inside one scale-up domain (tensor-parallel traffic stays on NVLink or Infinity Fabric), 400G per GPU for multi-node serving where requests, replicas and KV-cache movement share the network. It ignores KV-cache capacity for long contexts, quantization (FP8 or FP4 serving halves or quarters the weight footprint), and disaggregated prefill/decode architectures; treat it as a starting point.
What this estimator does not model: tensor-parallel and pipeline-parallel traffic (which can exceed data-parallel traffic for very large models), topology and oversubscription, storage and checkpoint traffic, latency, and vendor-specific collective algorithms. Real clusters are designed with all of these included. Use this tool to understand orders of magnitude and to sanity-check proposals — never as the design itself.
GPU reference specifications used by this estimator
Dense BF16 throughput is quoted without sparsity, because dense throughput is what the training math requires; vendor headline figures are often twice as high because they assume structured sparsity. Sources are vendor product pages and datasheets, listed in full under the table.
| GPU | Dense BF16 TFLOPS | Scale-up bandwidth per GPU | Scale-up domain | HBM per GPU | HBM bandwidth |
|---|---|---|---|---|---|
| NVIDIA H100 SXM | 989 | 900 GB/s (NVLink 4) | 8 GPUs / node | 80 GB HBM3 | 3.35 TB/s |
| NVIDIA H200 SXM | 989 | 900 GB/s (NVLink 4) | 8 GPUs / node | 141 GB HBM3e | 4.8 TB/s |
| NVIDIA B200 | 2,250 | 1,800 GB/s (NVLink 5) | 8 GPUs / node | 180 GB HBM3e | 8.0 TB/s |
| NVIDIA GB200 NVL72 (per GPU) | 2,500 | 1,800 GB/s (NVLink 5) | 72 GPUs / rack | 186 GB HBM3e | 8.0 TB/s |
| AMD Instinct MI300X | 1,307.4 | 896 GB/s (Infinity Fabric) | 8 GPUs / platform | 192 GB HBM3 | 5.3 TB/s |
| AMD Instinct MI325X | 1,307.4 | 896 GB/s (Infinity Fabric) | 8 GPUs / platform | 256 GB HBM3e | 6.0 TB/s |
Specification sources: NVIDIA H100 and H200 product/datasheet specifications (nvidia.com); NVIDIA Blackwell B200 datasheet figures as published by NVIDIA and summarized by Spheron and NextPCB; AMD Instinct MI300X platform data sheet (amd.com) and AMD Instinct MI325X product page (amd.com). Power-per-link figures: 800G DAC/AOC comparison data (Ascent Optics, QSFPTEK), 800G transceiver market data (Dataintelo), and Broadcom CPO energy figures summarized by Naddod and CableXpress. Full URLs are listed in the project README.
Embed this estimator on your site
This estimator is free to use on any website or blog — just paste the snippet below where you want it to appear. The only condition is that the built-in “Powered by OpenTenCor” attribution link stays in place; it is part of the widget and keeps the tool free for everyone.
Frequently asked questions
How much network bandwidth does LLM training need per GPU?
For data-parallel training, a useful first estimate comes from gradient synchronization: each GPU must exchange roughly 4 bytes per model parameter per step (a ring all-reduce factor of about 2, times 2 bytes per BF16 gradient). Multiply those bytes per step by steps per second and you get the cross-node bandwidth each GPU needs. On H100-class hardware at 40% MFU with 8,192 tokens per GPU per step, this works out to about 258 Gb/s per GPU, which is why 400G network adapters are the common baseline in modern training clusters and 800G is becoming the default for new builds.
Why does the training estimate barely change when I change the model size?
In this simplified data-parallel model, gradient traffic per step and compute per token both grow in direct proportion to the parameter count, so the parameter count cancels out of the division. Real clusters do get more network-hungry as models grow, because larger models need more tensor and pipeline parallelism, which adds traffic this estimator does not model. Treat the result as a floor for the data-parallel portion of the traffic, not a total.
When does an AI cluster need optical interconnects instead of copper?
Copper works inside the rack: GPU scale-up fabrics such as NVLink and Infinity Fabric run over backplanes and short cables, and passive direct-attach copper (DAC) reaches roughly 2 to 3 meters at 800G lane speeds. Traffic that leaves the scale-up domain — node to node and rack to rack — travels farther than copper can carry at these speeds, so those links are optical. In practice: within a node or rack, copper; beyond the rack, optics, today mostly pluggable transceivers and increasingly co-packaged optics (CPO).
What is the difference between scale-up and scale-out bandwidth?
Scale-up bandwidth is the GPU-to-GPU bandwidth inside a tightly coupled domain — one node, or one NVLink domain such as the 72-GPU domain of a GB200 NVL72 rack — provided by NVLink or Infinity Fabric at 900 GB/s to 1,800 GB/s per GPU. Scale-out bandwidth is the network bandwidth between nodes, provided by InfiniBand or Ethernet adapters at 200G, 400G or 800G per GPU. Scale-up carries tensor-parallel traffic; scale-out carries data-parallel gradient synchronization, pipeline traffic and storage traffic.
Is this estimator a substitute for a professional network design?
No. It is an educational planning aid that models one traffic type (data-parallel gradient synchronization) with stated assumptions and public vendor specifications. Real fabric design also depends on the parallelism strategy, topology, oversubscription ratio, storage traffic, checkpointing and vendor-specific tuning. Use the result to sanity-check quotes and to understand orders of magnitude, then validate with your infrastructure provider.