7.3 Word embeddings

What this is and why it exists

Word embeddings place words in a space where distance means relatedness, and they are the conceptual ancestor of every embedding you will use for search, retrieval and similarity. The famous demonstration — take the vector for king, subtract man, add woman, and the nearest word is queen — is worth being able to explain rather than recite, because the explanation tells you what these vectors are and, more usefully, what they are not.

The vocabulary

• Embedding — a word represented as a list of numbers, learned from data.
• Distributional hypothesis — the assumption that words in similar contexts have similar meanings.
• Context window — how many surrounding words count as context.
• Skip-gram — predicting the context from the word; CBOW — predicting the word from its context.
• Negative sampling — training against a few random non-context words instead of the whole vocabulary.
• Co-occurrence matrix — a count of how often each pair of words appears together.
• Static embedding — one fixed vector per word, regardless of use.
• Contextual embedding — a vector computed for a word in its particular sentence.

The mental model

Everything rests on one assumption: words that appear in similar contexts mean similar things. Nobody defines meaning; the training procedure only ever sees which words keep company with which. That is enough, and it is also the limit — the vectors encode statistical association in a corpus, which overlaps with meaning without being it.

The original approach learns vectors by making a prediction task work. In the skip-gram arrangement, a word is given and the model predicts the words around it; in the other, the surrounding words are given and the model predicts the middle one. In both cases the prediction is throwaway — what you keep is the vectors the model had to learn in order to predict well. Skip-gram handles rare words better because each rare occurrence is its own training example; the other arrangement is faster and smooths over rare words by averaging their context. Negative sampling is the trick that made it fast: rather than computing a score across the entire vocabulary at every step, score the true context word against a handful of random words, which turns an enormous classification into a small one.

The factorisation approach reaches similar vectors from the other direction, and the fact that it does is informative. Instead of predicting, count: build a matrix of how often each pair of words appears together across the corpus, then find vectors whose relationships reproduce those co-occurrence ratios. The prediction method learns from local windows one at a time; the counting method uses the whole corpus at once. Arriving at comparable representations by both routes is good evidence that the structure is in the language rather than in the algorithm.

Now the arithmetic, properly explained. Directions in the space come to correspond to relationships, because those relationships show up as consistent differences in context. The contexts distinguishing king from man are much the same as those distinguishing queen from woman — one is about royalty and applies equally to both pairs. So the difference between the first pair and the difference between the second pair point in roughly the same direction, and subtracting one and adding the other lands near the fourth word. The analogy works because the relationship is a consistent direction, not because the model reasons about kings.

Two honest caveats belong with the demonstration. The result is a nearest neighbour, not an equality — the point you land on is usually not exactly any word, and the original words are typically excluded from the search, which flatters the result. And analogies of this kind work well for some relations and poorly for others, so the celebrated examples are the ones that worked.

Subword vectors fix the vocabulary problem. Represent each word as the sum of vectors for its character fragments as well as itself, and a word never seen in training still gets a sensible vector built from familiar pieces. This matters far beyond convenience: for morphologically rich languages, where a single root generates dozens of inflected forms, a whole-word vocabulary either explodes or discards most of the forms, and either way many languages are served much worse than English. Fragment-based vectors handle that structure directly, and the same reasoning is why every modern model uses subword tokens.

Then the two limitations that motivate everything after this topic. First, one vector per word cannot represent more than one sense. The vector for bank is a compromise between the river and the money, sitting somewhere unhelpful between both; the sentence it appeared in is not consulted, because a static embedding never sees a sentence. Contextual models fixed exactly this by computing a word's vector from its surroundings each time, which is the direct line from here to the transformer topics.

Second, embeddings absorb the associations in their training text, including ones nobody wants a system acting on. Occupation words sit closer to one gender than another; names common in one community carry different associations than names common in another. This is not a flaw in the algorithm — it faithfully recorded what the corpus contained — and it is exactly why it is a deployment concern rather than a footnote. An embedding used for ranking résumés or retrieving documents propagates those associations into decisions, at scale and invisibly. Debiasing methods exist and are partial: they reduce the association along measured directions and do not remove the structure underneath. The reliable practices are to know what your embeddings were trained on, to test for the associations that would matter in your application, and to keep a human decision between an embedding and any consequential outcome.

What you should now be able to explain or do

State the distributional hypothesis and say what it does and does not capture. Describe both prediction arrangements and say which suits rare words. Explain negative sampling and the problem it solves. Say how the counting approach differs and why agreement between the two is informative. Explain the analogy result in terms of consistent directions, with both caveats. Say what subword vectors fix and which languages benefit most. Give the two limitations of static embeddings and what each led to.

Check yourself