Glossary of GPU Terms
GPU vocabulary makes real purchasing decisions, yet it is rarely defined plainly. This glossary explains the essential terms in plain English — VRAM, HBM, Tensor Cores, NVLink, MIG, spot instances, LoRA, quantization, KV cache, the GPU-hour, and more.
Shopping for GPU compute means wading through jargon. Here are the terms you will meet most often, explained without assuming a hardware background. For how these concepts affect pricing and GPU choice, see how GPU cloud pricing works and A100 vs H100 vs L40S.
GPU (Graphics Processing Unit)
Originally designed for rendering graphics, now the standard processor for AI workloads. GPUs excel at performing thousands of simple calculations in parallel — exactly what neural networks need.
VRAM (Video RAM)
The GPU's own dedicated memory. Model weights, activations, and optimizer states all live in VRAM during training and inference. If your model does not fit in VRAM, it cannot run on that GPU — which is why VRAM capacity is often the first spec to check.
HBM (High Bandwidth Memory)
A type of stacked memory used on data-center GPUs (HBM2e on the A100, HBM3 on the H100/H200). It delivers far higher bandwidth than standard GDDR memory, at higher cost. Bandwidth matters because AI workloads constantly shuttle data between memory and compute cores.
Memory bandwidth
How fast data moves between VRAM and the GPU's compute cores, measured in GB/s or TB/s. Many AI workloads are "memory-bound" — limited by bandwidth rather than raw compute — so bandwidth often predicts real performance better than FLOPS figures.
CUDA cores
NVIDIA's general-purpose parallel processing cores. They handle the bulk of non-specialized computation on the GPU. More CUDA cores generally means more parallel throughput, though architecture improvements matter too.
Tensor Cores
Specialized cores on modern NVIDIA GPUs designed to accelerate matrix multiplications — the core operation of neural networks. They deliver dramatically higher throughput than CUDA cores for AI math, especially at reduced precision (FP16, BF16, FP8, INT8).
TFLOPS
Trillions of floating-point operations per second — a measure of theoretical peak compute throughput. Useful for rough comparisons, but real workloads rarely achieve peak figures; memory bandwidth and software efficiency usually dominate.
FP32 / FP16 / BF16 / FP8
Floating-point precisions: 32-bit, 16-bit, "brain" 16-bit, and 8-bit. Lower precision uses less memory and runs faster on Tensor Cores. Modern training typically uses mixed precision (FP16/BF16 compute with FP32 master weights); inference increasingly uses FP8 or quantized integer formats.
INT8 / INT4 quantization
Representing model weights (and sometimes activations) as 8-bit or 4-bit integers instead of 16-bit floats. Quantization shrinks models dramatically — a 4-bit model uses roughly a quarter of the VRAM of its 16-bit version — with often small quality trade-offs for inference. See our cost-saving guide.
NVLink / NVSwitch
NVIDIA's high-bandwidth interconnect for GPU-to-GPU communication within a server. NVLink is roughly an order of magnitude faster than PCIe for GPU-to-GPU traffic; NVSwitch connects many GPUs (e.g. 8) together. Essential for tensor-parallel training of large models. See our multi-GPU guide.
PCIe (PCI Express)
The standard expansion-bus interface connecting GPUs to the rest of the system. Fine for single-GPU work and data-parallel multi-GPU setups, but much slower than NVLink for GPU-to-GPU communication.
SXM vs PCIe form factor
Two physical formats for data-center GPUs. SXM modules mount directly to the server board with NVLink connections (fastest multi-GPU); PCIe cards slot into standard PCIe slots (more flexible, slower GPU-to-GPU). The H100 comes in SXM and PCIe variants; the L40S is PCIe; the A100 comes in both.
MIG (Multi-Instance GPU)
A feature of the A100 and H100 that partitions one physical GPU into up to seven isolated instances, each with its own memory and compute. Useful for sharing a GPU across users or small jobs — or for running multiple experiments in parallel on one card.
TDP (Thermal Design Power)
The maximum heat a GPU is designed to dissipate, in watts — a proxy for power consumption. Higher TDP generally means more performance but also more electricity and cooling cost, which is part of why cloud GPU rates differ.
GPU-hour
The billing unit of GPU clouds: one GPU running for one hour. Four GPUs for six hours = 24 GPU-hours. Nearly all cloud GPU pricing is built on this unit. See how GPU cloud pricing works.
On-demand / spot / reserved
The three cloud pricing models. On-demand: standard rate, no commitment, always available (until sold out). Spot (preemptible): steep discount on spare capacity, can be reclaimed anytime. Reserved/committed: lower rate in exchange for a 1–3 year usage commitment.
Preemption
When a cloud provider reclaims a spot instance to give the capacity to someone else. Jobs must checkpoint and resume to survive preemption — the central skill of using spot instances well.
Checkpoint
A saved snapshot of training progress — model weights and usually optimizer state — written to persistent storage. Checkpoints let you resume after interruptions and are the reason spot instances are viable for training.
Epoch
One complete pass through the training dataset. Fine-tuning typically runs for a small number of epochs; pre-training is often a single pass over an enormous dataset.
Batch size
How many training examples are processed together in one step. Larger batches use more VRAM but improve GPU utilization; in data-parallel training, the effective batch size is the per-GPU batch times the number of GPUs.
Throughput (tokens/second)
How fast a setup processes data — the key number for estimating GPU hours. Measure it on your own workload; published figures are only starting points. See estimating GPU hours.
KV cache
Memory used during LLM inference to store previously computed key/value tensors, avoiding recomputation. It grows with sequence length and batch size and is a major VRAM consumer when serving long-context models.
LoRA / QLoRA
Parameter-efficient fine-tuning methods. LoRA freezes the base model and trains small adapter matrices; QLoRA adds 4-bit quantization of the frozen weights. Both slash the VRAM and compute needed for fine-tuning — the indie developer's best friend.
Data parallelism
Multi-GPU strategy where each GPU holds a full model copy and processes different data. Simple and effective when the model fits on one GPU. See our multi-GPU guide.
Tensor (model) parallelism
Multi-GPU strategy where the model itself is split across GPUs. Required when a model exceeds single-GPU VRAM; demands fast interconnects like NVLink.
Inference
Running a trained model to generate predictions or text — as opposed to training, which updates the model's weights. Inference is usually memory-capacity-bound and is where quantization pays off most.
Serverless inference
Inference you pay for per request or per token instead of per GPU-hour — the provider runs the model and scales to zero when idle. Convenient for sporadic traffic; the per-token price usually exceeds the raw GPU cost, so high-volume workloads are often cheaper on dedicated GPUs.
GPU marketplace / broker
A platform where independent hosts list spare GPUs for rent (a "Craigslist for GPUs"), or that aggregates many providers' prices in one place. Prices can be low, but availability, reliability, and support vary by host — unlike a single cloud with SLAs.
Egress
Data transferred out of a cloud provider's network, commonly billed per gigabyte. A frequent surprise line item for developers who download large models or datasets repeatedly.
Four more terms worth knowing
Spot / preemptible instance
Discounted cloud capacity that the provider can reclaim on short notice. Cheap for work that can pause and resume, risky for anything with a hard deadline. See the guide to spot instances and checkpointing.
Checkpoint
A saved snapshot of a training job's state - model weights, optimizer state and progress counters - written to storage so an interrupted job can resume instead of restarting.
Egress
Data transferred out of a cloud provider's network, commonly billed per gigabyte. Downloading large models or datasets repeatedly makes this a frequent surprise line item.
Quantization
Storing model weights at lower numerical precision, such as 8-bit or 4-bit integers instead of 16-bit floats, to reduce memory use and cost in exchange for a small, measurable change in output quality.