5.20 Model interpretability

What this is and why it exists

Sooner or later somebody asks why the model said that, and "the gradient boosting decided" is not an answer. In regulated and high-stakes settings it is a shipping requirement rather than a courtesy, and even where nobody demands it, explanations are the fastest way to find out that your model has learned something absurd. This topic is the standard tools and — more importantly — exactly what each of them licenses you to say.

The vocabulary

• Global explanation — what the model relies on overall.
• Local explanation — why this one prediction came out as it did.
• Permutation importance — shuffle a feature, re-score, and measure the loss.
• Partial dependence — the model's average prediction as one feature is varied across its range.
• ICE curve — the same for one individual row rather than averaged.
• Shapley value — a way of dividing credit for a prediction among the features, borrowed from game theory.
• Additive attribution — the property that the contributions plus a baseline add up to the prediction.
• Faithfulness — whether the explanation describes the model's actual behaviour.

The mental model

Two questions with different answers. Global: across all predictions, what does this model lean on? Local: for this application, this transaction, this student, what drove the number? Regulators usually want the second, debugging usually wants the first, and a stakeholder asking "is this model sensible" wants the first and is reassured by the second.

Permutation importance answers the global question by damage: shuffle one feature's values so it carries no information, re-score on held-out data, and see how much worse the model does. What it measures is what the model actually relies on — which is the right question and better than the impurity-based default from the forests topic, which measures how often a feature was chosen. Its two limitations were named there and are worth repeating: correlated features share information, so shuffling either alone loses little and both look unimportant; and shuffling creates rows that could never occur, so the model is being evaluated somewhere it has never been, which can distort the result. Assess correlated features as a group, and read the numbers as approximate.

Partial dependence answers the shape question: as this feature increases, what happens to the prediction on average? It is the plot that tells you the model learned a sensible relationship — that risk rises with the amount, that the effect of practice flattens after a while — and it is the one most likely to expose nonsense, such as a relationship pointing the wrong way. Averaging is also its weakness: if the feature helps half the population and hurts the other half, the average is flat and the plot says nothing. ICE curves fix that by drawing one line per row instead of the average, so a fan of lines going in opposite directions is immediately visible. Draw both; the pair costs nothing extra and the disagreement between them is informative.

Shapley values answer the local question, and the idea is worth stating properly because it is what makes them trustworthy. Consider every possible order in which features could be added to the prediction, and average how much each feature changes the outcome when it joins. That average is that feature's contribution, and it has the property that the contributions plus a baseline add up exactly to the prediction — which is why they can be shown as a per-prediction breakdown. Computing this exactly is infeasible for many features, so the implementations approximate, with model-specific shortcuts that are fast and exact for tree ensembles.

Then the caution this topic ends on, and it is the same one the linear-regression topic made. An explanation attributes the model's behaviour, not the world's. A large contribution means the model used that feature, not that the feature causes the outcome. With correlated features, the credit can be split between them or swapped from one to another by a small change in the data, so a specific attribution should not be read as a specific mechanism. The honest sentence to a stakeholder is "the model weighted this heavily, in this case, and here is what happens if it changes" — and if they want a causal answer, that is a different study with a different design.

Two practical habits close it out. Explain on held-out data, not on the training set, or you are describing what the model memorised. And use explanations as a debugging instrument before using them as a report: a feature at the top of the importance list that nobody expected is one of the two most useful signals in applied machine learning, because it is either a genuine discovery or — far more often — the leakage the cleaning topic warned about, announcing itself.

What you should now be able to explain or do

Distinguish global from local explanation and say which each audience wants. Describe permutation importance and its two limitations. Say what partial dependence shows, when averaging hides the truth, and what ICE curves add. Explain Shapley values in terms of averaging over orders, and say what the additive property makes possible. State exactly what an explanation licenses you to say and what it does not. Say why an unexpected top feature is worth investigating immediately.

Check yourself