6.17 Attention from first principles
You can write scaled dot-product attention from memory.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Attention is retrieval: queries scored against keys, softmax weights over values, scaled to keep gradients healthy; multi-head runs several lookups in parallel, and masking enforces causality and padding. Writing it from memory once is the point of this topic — after that the transformer stops being a diagram and becomes a mechanism. It sits here as the hinge between the sequence models before it and the language-model world after.
Work through these
Query, key, value and the retrieval analogy
Attention is a lookup where every position asks a question, every position advertises what it holds, and the answer is a weighted blend of contents. That retrieval analogy carries most of the intuition.
Scaled dot-product attention and the sqrt(d) term
The specific formulation used everywhere, including the division that keeps the scores from growing with dimension. That scaling term exists for a concrete numerical reason worth understanding.
Multi-head attention
Running several attention operations in parallel lets the model attend to different kinds of relationship at once. It is a small change with a large effect on capacity.
Masking: padding and causal
Hiding padding so it is never attended to, and hiding the future so a model cannot read ahead. The second is what makes generation possible.
Sign in to keep your progress.
Free resources
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.