5.1 Framing an ML problem

What this is and why it exists

Most failed machine-learning projects failed before any code was written. They failed at framing: nobody said precisely what was being predicted, for which unit, against what alternative, or whether a model was warranted at all. This lesson is that step — the half-hour of thinking that decides whether the following three months are worth anything.

The vocabulary

• Supervised learning — learning from examples with known answers.
• Unsupervised learning — finding structure with no answers given.
• Self-supervised learning — creating the answers from the data itself, by hiding part of it and predicting it.
• Reinforcement learning — learning from the consequences of actions rather than from labelled examples.
• Target — the thing being predicted, defined precisely enough that two people would label the same row identically.
• Unit of prediction — the thing one row is about: a student, a session, a student-day, a transaction.
• Baseline — the simplest thing that could work, which your model must beat to be worth anything.
• Decision — what somebody will do differently because of the prediction.

The mental model

Start at the end. What decision changes because of this prediction, and who makes it? If nobody acts differently, the model is a report at best. This one question kills more bad projects than any technical review, and it also settles most of the others: how accurate is accurate enough, when is a prediction needed, and what a mistake costs in each direction.

Then define the target exactly. "Predict churn" is not a target; "predict whether an active learner will have no session for 28 consecutive days, as measured 28 days after the prediction date" is. The test is that two people given the definition would label the same row the same way. Vagueness here does not stay vague — it becomes an arbitrary choice made silently while writing the query, and nobody can reproduce or debate it later.

Then the unit of prediction, which is where the row count comes from and where leakage often enters. One row per student is a different problem from one row per student per week, and mixing them produces both duplicate information and a validation split that separates rows but not students. Say what one row means before writing the query.

Then the baseline, and this is the step people skip. Beat something you could have shipped this afternoon. For a forecast, "tomorrow equals today" or "this week equals the same week last year". For a classifier, always predicting the majority class, or a single sensible rule a domain expert would write. For a recommender, "the most popular thing". These are often startlingly good, and knowing that early is valuable in both directions: if your model beats the baseline by a little, the business wanted the rule; if it beats it by a lot, you have a number that justifies the work to anybody who asks.

Which leads to the honest question the topic ends on: when is machine learning the wrong tool? When the rule is already known and can be written down outright. When you have no labels and no way to get them. When you have too few examples for the pattern's complexity. When the cost of being wrong is high and nobody can accept a model that is right most of the time. When the thing you would predict is not actually actionable. And when the data you would need at prediction time is not available at prediction time — a spectacularly common discovery made three weeks in, when a feature that looked wonderful turns out to be filled in by a human after the outcome is already known.

Finally, the four learning shapes, which follow from what you have rather than from taste. Labels for what you want to predict: supervised. No labels and a structural question: unsupervised. Lots of unlabelled data and a way to make a prediction task out of it — hiding the next word, hiding a patch of an image: self-supervised, which is how modern language and vision models are trained. Sequential decisions with delayed consequences: reinforcement learning, which is a much bigger commitment than it looks and rarely the right first answer.

What you should now be able to explain or do

Ask the decision question of a proposed project and say what it settles. Write a target definition two people would apply identically. State the unit of prediction and say what it means for the split. Name a baseline you could ship this afternoon for three different problems. Give four situations in which machine learning is the wrong tool. Match a described problem to one of the four learning shapes, from what data exists rather than from preference.

Check yourself