7.2 Bag of words, TF-IDF and n-grams

What this is and why it exists

Counting words and putting the counts into a linear model is a text classifier you can build before lunch, and on a great many real tasks it is embarrassingly hard to beat. That is the first reason for this topic. The second is that without this baseline you cannot say whether anything larger bought you anything — you have a number and no reference point. And the third is that the sparsity you meet here is the problem embeddings were invented to solve, so meeting it properly makes the next topic land.

The vocabulary

• Document — one unit of text being classified.
• Corpus — the collection of documents.
• Bag of words — a representation counting which words occur, ignoring their order.
• Sparse matrix — one that is mostly zeros, stored by its non-zero entries only.
• Term frequency — how often a word occurs in a document.
• Inverse document frequency — a weight that falls as a word appears in more documents.
• n-gram — a sequence of n adjacent words or characters.
• Feature explosion — the rapid growth in feature count as n grows.

The mental model

Start from the problem it solves. The scikit-learn guide states it directly: raw text "cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length". The bag-of-words answer is to fix a vocabulary, give each word a column, and fill in how many times it occurred — so "documents are described by word occurrences while completely ignoring the relative position information of the words in the document".

That last clause is the whole assumption and it should feel wrong. "The film was not good" and "the film was good, not" produce nearly the same vector. And yet the representation works, because for most classification tasks the words present carry most of the signal and the order carries a smaller part. Knowing exactly what the representation threw away is what lets you predict where it will fail — negation, sarcasm, and anything where the same words in a different arrangement mean something else.

Sparsity is the defining property, and the guide is precise about its scale: "as most documents will typically use a very small subset of the words used in the corpus, the resulting matrix will have many feature values that are zeros (typically more than 99% of them)". Its example makes it concrete — ten thousand short documents draw on a vocabulary "in the order of 100,000 unique words in total while each document will use 100 to 1000 unique words individually".

Two consequences follow. Store it as a sparse matrix, which keeps only the non-zero entries; converting to a dense array is the standard way to exhaust memory on a dataset that would otherwise be comfortable. And prefer methods that handle sparse high-dimensional input well — linear models above all, which is why a linear model on counts is the classic pairing. Distance-based methods, by contrast, suffer badly here, for the reasons the classical module gave.

Weighting fixes the obvious flaw in raw counts. The guide names it: "some words will be very present (e.g. 'the', 'a', 'is' in English) hence carrying very little meaningful information about the actual contents of the document. If we were to feed the direct count data directly to a classifier those very frequent terms would shadow the frequencies of rarer yet more interesting terms." So the counts are re-weighted — "it is very common to use the tf-idf transform" — multiplying how often a word appears in this document by a factor that falls as the word appears in more documents across the corpus. A word common here and rare elsewhere scores high; a word common everywhere scores near nothing. It is a small adjustment that improves nearly every count-based method, and it removes most of the need to strip stopwords by hand, since they are down-weighted automatically.

n-grams recover a little of the order you discarded. Counting adjacent pairs as well as single words lets "not good" be its own feature, which repairs the negation failure directly. The cost is growth: a vocabulary of fifty thousand words admits far more possible pairs, and while only a fraction occur, the feature count still multiplies. Triples are worse and rarely worth it on ordinary data.

Three controls keep this manageable, and they are the practical craft of the topic. Set a minimum document frequency so a pair occurring twice in the corpus never becomes a feature — this alone removes most of the growth and loses almost nothing. Set a maximum document frequency to drop terms appearing nearly everywhere. And cap the number of features outright, keeping the most frequent. Character n-grams deserve a mention too: counting sequences of characters rather than words is robust to misspellings and to languages that do not separate words with spaces, and it frequently outperforms word n-grams on short, noisy text.

So build the baseline, and build it first. Weighted counts with single words and pairs, a minimum document frequency, and a linear model. It trains in seconds, it needs no accelerator, its errors are inspectable — you can read the weights and see which words drove a decision — and it produces the number every later approach must beat. When you later report that a large model reached some accuracy, the first question anybody competent will ask is what the simple baseline got, and not having that number is the difference between a result and an anecdote.

What you should now be able to explain or do

State the problem the representation solves and the assumption it makes. Predict where ignoring word order will fail. Explain sparsity, its scale, and the two consequences for storage and method choice. Say what weighting does and which problem it fixes. Use n-grams to recover order, and control the resulting growth with the three settings. Say when character n-grams beat word n-grams. Build the baseline and explain why reporting it is not optional.

Check yourself