6.7 Normalization layers
You know which normalization a given architecture needs.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Normalisation layers keep activations in trainable ranges, and the choice is architectural, not aesthetic: batch norm for convnets, layer norm for transformers, with batch-size sensitivity explaining several mysteries. It sits in the craft sequence because deep stacks barely train without it. The dual-behaviour trap is the classic: batch norm acts differently in training and evaluation, and a forgotten eval call at inference produces bugs that look like anything but what they are.
Work through these
Batch normalization: train vs. eval behaviour
Normalising across the batch stabilises training and behaves differently during training and inference, which is a common source of confusing bugs. Understanding the two modes is essential.
Layer norm and why transformers use it
Normalising across features instead of across the batch removes the dependence on batch size, which is why sequence architectures use it. This is the practical difference that decided the convention.
RMSNorm, GroupNorm, InstanceNorm
Three further variants, each normalising over a different grouping for a different architecture. Knowing which one a paper is using saves misreading its results.
Normalization and batch-size sensitivity
Some normalisation choices make results depend on batch size in ways that break when you change hardware. This is why reproducing a result at a different scale can be difficult.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.