8.6 Embeddings and vector databases

What this is and why it exists

Embeddings turn text into vectors whose closeness reflects meaning, and a vector index makes millions of them searchable in milliseconds. That is the retrieval half of every grounded system, and it is useful on its own — semantic search over your own documents, where a query finds the right passage even though it shares no words with it. This topic also carries the failure that breaks search invisibly: vectors from different models live in unrelated spaces, and comparing them produces numbers that mean nothing at all.

The vocabulary

• Embedding — a vector representing a piece of text, produced by a model.
• Dimensionality — how many numbers are in that vector.
• Cosine similarity — closeness measured by the angle between two vectors.
• Normalisation — scaling a vector to unit length.
• Approximate nearest neighbour — finding close vectors without checking every one.
• HNSW — a multilayer graph index.
• Recall — the share of the true nearest neighbours an approximate search returns.
• Hybrid retrieval — combining keyword scoring with vector similarity.

The mental model

An embedding model maps text to a point in a space where nearby means related. Everything follows from that: search is a nearest-neighbour query, clustering is clustering, deduplication is a distance threshold, and "find more like this" is free.

Choosing a model is the first decision, and three properties decide it. Dimensionality trades quality against storage and speed — more numbers hold more distinction and cost proportionally more memory and index time. Maximum input length decides your chunk size, since text beyond it is silently truncated by many wrappers, and a chunk whose ending vanished is a chunk whose vector describes something else. And domain fit: a model trained on general web text will be mediocre on legal, clinical or heavily technical material, and the way to know is to evaluate on your own queries rather than on a published average.

Then the failure that gives this topic its warning. Vectors from different models are not comparable. Each model learned its own space, and its dimensions mean nothing outside it. Two vectors of the same length from two models are two unrelated points, and the similarity between them is a number with no meaning — not a bad score, a meaningless one. So: embed your documents and your queries with the same model; re-embed everything when you change model or version; and record which model produced each vector alongside the vector, because a store containing vectors from two models silently returns nonsense for some queries and plausible results for others, which is the worst possible failure shape.

Related and equally quiet: normalise before comparing by cosine. Cosine similarity is about direction and ignores magnitude, so if your store computes an inner product instead — which is faster and equivalent only for unit-length vectors — unnormalised vectors let long documents dominate purely by having larger magnitudes. Some models return normalised vectors and some do not. Check, rather than assume.

Approximate search is what makes this scale. Comparing a query against every vector is exact and linear in the collection, which is fine for thousands and hopeless for millions. Approximate indexes trade a little recall for an enormous speed gain, and the trade is nearly always worth taking, because the retrieved passages then go to a model that is itself imprecise — losing one borderline neighbour in twenty changes almost nothing.

The pgvector documentation states the two main index families' trade plainly. An HNSW index "creates a multilayer graph. It has better query performance than IVFFlat (in terms of speed-recall tradeoff), but has slower build times and uses more memory." An IVFFlat index "divides vectors into lists, and then searches a subset of those lists that are closest to the query vector. It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff)." So: the graph index when queries matter more than build cost, which is most read-heavy systems; the list index when the collection changes constantly or memory is tight.

On where to put the vectors: start with the database you already run. The vector extension for PostgreSQL adds a column type — CREATE TABLE items (id bigserial PRIMARY KEY, embedding vector(3)); — and distance operators, of which <=> is cosine distance, <-> is L2, <#> is negative inner product and <+> is L1. That means your vectors sit beside your ordinary data, in the same transaction, with the same backup and the same access control, and a filter on a normal column and a similarity ordering are one query.

That last point is worth more than it sounds, and it is why the alternatives need a reason. In-process libraries are fast and give you the persistence and concurrency problem to solve yourself. Purpose-built services handle sharding, replication and very large collections, and add an operational component with its own consistency story. They earn their place at scale or with demanding filtering; below that, one fewer system to run and back up is the stronger argument.

Metadata filtering is not a detail. Real queries are almost never pure similarity — they are similarity within this customer's documents, from the last year, of this type, that this user may see. Two orders exist: filter first and search the survivors, which is right when the filter is very selective; or search first and filter the results, which is right when it is not, and which can return too few rows if the filter removes most of them. A store that does this well plans it for you; one that does not makes you choose, and choosing wrong is a common cause of empty or slow results. Access control belongs in the filter, not in the prompt — retrieval that returns documents a user may not see has already leaked them, whatever the model says afterwards.

Finally, pure vector search alone underperforms on many real queries, which surprises people who have only seen it work beautifully on paraphrase. It is weak exactly where keyword search is strong: exact identifiers, product codes, error numbers, rare proper nouns, and any term the embedding model never saw. Combining a classical keyword score with vector similarity — running both and merging the ranked lists — is usually the single biggest improvement available, and it is the first thing the advanced retrieval topic reaches for.

What you should now be able to explain or do

Say what an embedding is and what its three selection properties trade. Explain why vectors from different models cannot be compared and what to record and re-embed. Say when normalisation matters and why. Explain what approximate search trades and why the trade is acceptable here. Choose between the two index families using their documented properties. Argue for starting with the database you already run, and say what would justify a dedicated store. Combine metadata filtering with similarity, in the right order, and put access control in the filter. Say where pure vector search is weak.

Check yourself