Glossary of GPU Terms
Shopping for GPU compute means wading through jargon. Here are the terms you will meet most often, explained without assuming a hardware background. For how these concepts affect pricing and GPU choice, see how GPU cloud pricing works and A100 vs H100 vs L40S.
GPU (Graphics Processing Unit)
Originally designed for rendering graphics, now the standard processor for AI workloads. GPUs excel at performing thousands of simple calculations in parallel — exactly what neural networks need.
VRAM (Video RAM)
The GPU's own dedicated memory. Model weights, activations, and optimizer states all live in VRAM during training and inference. If your model does not fit in VRAM, it cannot run on that GPU — which is why VRAM capacity is often the first spec to check.
HBM (High Bandwidth Memory)
A type of stacked memory used on data-center GPUs (HBM2e on the A100, HBM3 on the H100). It delivers far higher bandwidth than standard GDDR memory, at higher cost. Bandwidth matters because AI workloads constantly shuttle data between memory and compute cores.
Memory bandwidth
How fast data moves between VRAM and the GPU's compute cores, measured in GB/s or TB/s. Many AI workloads are "memory-bound" — limited by bandwidth rather than raw compute — so bandwidth often predicts real performance better than FLOPS figures.
CUDA cores
NVIDIA's general-purpose parallel processing cores. They handle the bulk of non-specialized computation on the GPU. More CUDA cores generally means more parallel throughput, though architecture improvements matter too.
Tensor Cores
Specialized cores on modern NVIDIA GPUs designed to accelerate matrix multiplications — the core operation of neural networks. They deliver dramatically higher throughput than CUDA cores for AI math, especially at reduced precision (FP16, BF16, FP8, INT8).
TFLOPS
Trillions of floating-point operations per second — a measure of theoretical peak compute throughput. Useful for rough comparisons, but real workloads rarely achieve peak figures; memory bandwidth and software efficiency usually dominate.
FP32 / FP16 / BF16 / FP8
Floating-point precisions: 32-bit, 16-bit, "brain" 16-bit, and 8-bit. Lower precision uses less memory and runs faster on Tensor Cores. Modern training typically uses mixed precision (FP16/BF16 compute with FP32 master weights); inference increasingly uses FP8 or quantized integer formats.
INT8 / INT4 quantization
Representing model weights (and sometimes activations) as 8-bit or 4-bit integers instead of 16-bit floats. Quantization shrinks models dramatically — a 4-bit model uses roughly a quarter of the VRAM of its 16-bit version — with often small quality trade-offs for inference. See our cost-saving guide.
NVLink / NVSwitch
NVIDIA's high-bandwidth interconnect for GPU-to-GPU communication within a server. NVLink is roughly an order of magnitude faster than PCIe for GPU-to-GPU traffic; NVSwitch connects many GPUs (e.g. 8) together. Essential for tensor-parallel training of large models. See our multi-GPU guide.
PCIe (PCI Express)
The standard expansion-bus interface connecting GPUs to the rest of the system. Fine for single-GPU work and data-parallel multi-GPU setups, but much slower than NVLink for GPU-to-GPU communication.
SXM vs PCIe form factor
Two physical formats for data-center GPUs. SXM modules mount directly to the server board with NVLink connections (fastest multi-GPU); PCIe cards slot into standard PCIe slots (more flexible, slower GPU-to-GPU). The H100 is primarily SXM; the L40S is PCIe; the A100 comes in both.
MIG (Multi-Instance GPU)
A feature of the A100 (and H100) that partitions one physical GPU into up to seven isolated instances, each with its own memory and compute. Useful for sharing a GPU across users or small jobs — or for running multiple experiments in parallel on one card.
TDP (Thermal Design Power)
The maximum heat a GPU is designed to dissipate, in watts — a proxy for power consumption. Higher TDP generally means more performance but also more electricity and cooling cost, which is part of why cloud GPU rates differ.
GPU-hour
The billing unit of GPU clouds: one GPU running for one hour. Four GPUs for six hours = 24 GPU-hours. Nearly all cloud GPU pricing is built on this unit. See how GPU cloud pricing works.
On-demand / spot / reserved
The three cloud pricing models. On-demand: standard rate, no commitment, always available (until sold out). Spot (preemptible): steep discount on spare capacity, can be reclaimed anytime. Reserved/committed: lower rate in exchange for a 1–3 year usage commitment.
Preemption
When a cloud provider reclaims a spot instance to give the capacity to someone else. Jobs must checkpoint and resume to survive preemption — the central skill of using spot instances well.
Checkpoint
A saved snapshot of training progress — model weights and usually optimizer state — written to persistent storage. Checkpoints let you resume after interruptions and are the reason spot instances are viable for training.
Epoch
One complete pass through the training dataset. Fine-tuning typically runs for a small number of epochs; pre-training is often a single pass over an enormous dataset.
Batch size
How many training examples are processed together in one step. Larger batches use more VRAM but improve GPU utilization; in data-parallel training, the effective batch size is the per-GPU batch times the number of GPUs.
Throughput (tokens/second)
How fast a setup processes data — the key number for estimating GPU hours. Measure it on your own workload; published figures are only starting points. See estimating GPU hours.
KV cache
Memory used during LLM inference to store previously computed key/value tensors, avoiding recomputation. It grows with sequence length and batch size and is a major VRAM consumer when serving long-context models.
LoRA / QLoRA
Parameter-efficient fine-tuning methods. LoRA freezes the base model and trains small adapter matrices; QLoRA adds 4-bit quantization of the frozen weights. Both slash the VRAM and compute needed for fine-tuning — the indie developer's best friend.
Data parallelism
Multi-GPU strategy where each GPU holds a full model copy and processes different data. Simple and effective when the model fits on one GPU. See our multi-GPU guide.
Tensor (model) parallelism
Multi-GPU strategy where the model itself is split across GPUs. Required when a model exceeds single-GPU VRAM; demands fast interconnects like NVLink.
Inference
Running a trained model to generate predictions or text — as opposed to training, which updates the model's weights. Inference is usually memory-capacity-bound and is where quantization pays off most.
Egress
Data transferred out of a cloud provider's network, commonly billed per gigabyte. A frequent surprise line item for developers who download large models or datasets repeatedly.