5.2 Linear regression from scratch

What this is and why it exists

Linear regression is the first complete model: a way of predicting, a way of measuring how wrong you are, a way of fitting, and a way of interpreting the result. Implementing it once without a library sets the template every later model follows, including neural networks — which are this, stacked, with a nonlinearity between the layers.

The vocabulary

• Model — the prediction is a weighted sum of the features plus an intercept.
• Weight (coefficient) — how much one feature moves the prediction.
• Intercept — the prediction when every feature is zero.
• Loss — the number measuring how wrong the model is; here, the mean squared error.
• Closed form — solving for the best weights directly with linear algebra.
• Gradient descent — stepping downhill on the loss surface until it stops improving.
• Residual — the actual value minus the predicted one.
• Basis expansion — adding transformed features so a linear model can fit a curve.

The mental model

The model in words: multiply each feature by its weight, add them up, add the intercept, and that is the prediction. The loss in words: take each residual, square it, and average — squaring makes errors positive and punishes large ones disproportionately, which is a choice with consequences and not a law of nature.

There are two ways to find the best weights, and knowing both is the point of the topic rather than an academic exercise.

The closed form solves it in one step. Because the squared-error loss is a bowl with exactly one lowest point, calculus gives the answer directly: the weights are the inverse of the feature matrix multiplied by its own transpose, times that transpose, times the targets — the normal equations. It is exact and it stops being practical when the features are many, because inverting a large matrix is expensive and becomes unstable when features are nearly duplicates of each other.

Gradient descent walks downhill instead. Start anywhere, compute the slope of the loss with respect to each weight, step a little in the opposite direction, repeat. For the squared-error loss the gradient has a form worth saying aloud: for each weight, it is minus two over n, times the sum over all rows of that row's residual times that row's value of the feature. Read it as each feature is nudged in proportion to how much it co-varies with the error you are still making. That sentence is the whole of gradient-based learning, and everything in the deep-learning module is this with more layers.

The step size is the one setting that decides whether this works: too small and it takes forever, too large and it overshoots and diverges. And because the size of the gradient depends on the size of the feature, unscaled features make one direction of the loss surface far steeper than another — which is why scaling matters for gradient methods and not for the closed form.

Then the assumptions, which are worth naming rather than reciting. The relationship is linear in the parameters; the residuals are independent, have roughly constant spread, and are not systematically related to the predictions. You check these by looking, not by testing: plot the residuals against the predictions. A curve in that plot means the linearity assumption is failing; a funnel means the spread grows with the prediction; a pattern over time means the independence assumption is broken. Each has a fix — a transformation, a different loss, a model of the time structure — and all of them start with the plot.

Then the trap, which matters more than everything above because it changes what you are allowed to say. A coefficient describes association within this dataset and this feature set. It is not a causal effect. Add a correlated feature and the coefficients redistribute between them; drop one and the other absorbs its role. With two features that move together, the individual weights can be large, unstable and opposite in sign while the predictions stay fine. So the honest reading is "holding the other features in this model fixed, an increase of one unit is associated with this much change, in this data" — and the moment somebody hears a coefficient as "if we do more of this, that will happen", say so out loud.

Basis expansion is the escape from strict straight lines. Add the square of a feature, or its log, or a product of two features, and the model is still linear in its parameters while the curve it draws is not. That is the trick that makes linear models far more capable than they first appear — and the point at which you must start watching for overfitting, which is the next two topics.

What you should now be able to explain or do

Write the model, the loss and the gradient in words, without notation. Say why the closed form is exact and when it stops being practical. Explain what the gradient means in one sentence about co-varying with the error. Say why the step size and feature scaling matter for gradient descent and not for the closed form. Read a residual plot and name what each pattern indicates. State honestly what a coefficient does and does not license you to say. Fit a curve with a linear model and explain why it is still linear.

Check yourself