6.21 Scaling training: precision, accumulation, distribution
You can train a model larger than one GPU's memory.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Training beyond one GPU's memory is engineering, not magic: mixed precision halves memory, gradient accumulation fakes large batches, and DDP plus FSDP or ZeRO split work and optimizer states across devices. It closes the deep learning module because it scales everything the module built. The economics deserve scepticism — more GPUs do not mean proportional speed, and an unprofiled input pipeline can leave expensive hardware idling in perfect synchrony.
Work through these
Mixed precision (fp16/bf16) and loss scaling
Using lower-precision numbers for most of the computation, with safeguards that stop small gradients disappearing. It roughly halves memory and speeds up training substantially.
Gradient accumulation and effective batch size
Accumulating gradients over several small batches before stepping simulates a large batch on small hardware. It is the simplest way to train something that will not otherwise fit.
Data parallel, DDP, FSDP/ZeRO
Splitting training across several accelerators, from simple replication to schemes that also split the model's own state. The more advanced schemes are what make very large models trainable at all.
Profiling GPU utilization and the input pipeline
Accelerators frequently sit idle waiting for data, and measuring that is the first step in making training faster. Optimising the model before checking this is a common mistake.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.