3.8 Feature engineering

What this is and why it exists

Feature engineering is where what you know about the problem enters the model, and it is routinely worth more than a better algorithm. A well-chosen feature can turn a mediocre model into a good one; no amount of tuning rescues a model that cannot see the thing that matters. This topic is the standard toolkit — encodings, scalings, decompositions of dates and text — plus the one technique that leaks the answer into your features if you use it carelessly.

The vocabulary

• Feature — a column the model actually sees.
• One-hot encoding — one indicator column per category.
• Ordinal encoding — categories mapped to numbers, which asserts an order.
• Target encoding — a category replaced by the average target for that category.
• Hashing — mapping categories to a fixed number of columns by hash, so unseen values need no new column.
• Scaling — putting features on comparable ranges.
• Interaction — a feature combining two others, because their effect together is not the sum of their effects apart.
• Binning — turning a continuous value into ranges.

The mental model

Encoding is about telling the truth to the model. One-hot asserts nothing beyond "these are different", which is right for unordered categories — subject, region, device. Ordinal asserts an order and a spacing, so it is right for genuinely ordered things — low, medium, high — and wrong for anything else, because numbering four cities one to four tells a linear model that the third is three times the first. That mistake is common and completely silent.

The cost of one-hot is width: a column with a thousand distinct values becomes a thousand columns, most of them almost always zero. Two ways out. Group the long tail into an "other" bucket, which is usually enough and keeps the result readable. Or use hashing, which maps categories into a fixed number of columns and therefore handles values it has never seen — useful at genuinely high cardinality, at the price of collisions and of features nobody can interpret.

Target encoding is the powerful one and the dangerous one. Replacing a category with the average target for that category compresses a thousand categories into one informative number, and it directly encodes the label into a feature. Computed over the whole dataset it is leakage of the purest kind — each row's feature contains the row's own answer — and the validation score that results is fiction that evaporates on real data. If you use target encoding, compute it out-of-fold: for each fold, the encoding is learned from the other folds only, and the mapping applied to validation and test comes from training data alone. Add smoothing towards the overall mean so that a category with three examples does not get a confident value. If that sounds fiddly, the honest alternative is to use one-hot with a grouped tail and spend the effort elsewhere.

Scaling matters for some model families and not others, and knowing which saves pointless work. Anything using distances or gradients cares — nearest neighbours, support vector machines, linear models with regularisation, neural networks — because a feature measured in thousands otherwise dominates one measured in units. Tree-based models do not care at all, since they split on order rather than magnitude. Transformations are the neighbouring tool: a heavily skewed positive quantity often behaves much better after a log, and the reason is worth understanding rather than copying — it makes multiplicative differences additive, which is the shape most linear models expect.

Decomposition is where domain knowledge pays. A timestamp is a poor feature and its parts are excellent ones: hour of day, day of week, whether it is a weekend, whether it is a holiday, days since the previous event. Cyclical values need care — hour 23 and hour 0 are adjacent, and encoding them as 23 and 0 puts them at opposite ends — which is what the sine-and-cosine pair is for. Text decomposes into length, counts, presence of specific markers before any model of language is involved. Geography decomposes into distance from a point of interest, region, and density.

Interactions and binning are the last two, and both should be deliberate. An interaction says the effect of one feature depends on another — a discount matters more on an expensive item — and a linear model cannot discover that on its own while a tree model largely can, so add them where you have a reason. Binning throws away information in exchange for robustness and interpretability; it is often right for a report and usually wrong for a model, which can generally use the continuous value better than your chosen boundaries can.

Above all, the constraint the previous topic established still governs everything here: a feature may contain only what is knowable at prediction time, and every transformation is fitted on training data and applied to the rest.

What you should now be able to explain or do

Choose an encoding for four described columns and say what each encoding asserts. Handle a high-cardinality column two ways and give the cost of each. Say exactly why naive target encoding leaks, and describe the out-of-fold procedure that fixes it. Say which model families need scaling and which do not, and why. Decompose a timestamp into useful features, including the cyclical ones. Say when an interaction is worth adding and when binning is the wrong move.

Check yourself