EC-18.6 Functions of Several Variables, and Partial Derivatives

What this is and why it exists

Nothing in engineering depends on one quantity alone.

A gain depends on a resistor and a supply voltage. A field depends on three coordinates. A power loss depends on current, temperature and frequency at once. Every one of those is a function of several variables, and the single-variable derivative has nothing to say about them.

This topic is where the derivative learns to handle that. It is also where a reader arriving from the machine learning side of this site meets the gradient again. There it is the thing an optimiser walks down. Here it is the thing that says which way a surface rises.

The vocabulary

• Function of several variables — a rule taking two or more inputs to one output.
• Level curve — the set of inputs giving one fixed output, drawn in the plane.
• Partial derivative — the rate of change due to one variable while the others are held fixed.
• Mixed partial derivative — a second derivative taken with respect to two different variables.
• Differentiable — well approximated near a position by a linear function of the inputs.
• Total differential — the expression giving the change in output when every input moves a little.
• Gradient — the collection of first partial derivatives, read as a direction.
• Directional derivative — the rate of change along a chosen direction.
• Saddle — a stationary position that is a maximum one way and a minimum another.
• Lagrange multiplier — the method for optimising subject to a constraint.

The mental model

Start with the picture, because it stops being available almost immediately. A function of two variables is a surface above the plane. A function of three variables is not a picture at all.

So the practical tool is the level curve: hold the output fixed and draw the set of inputs that give it. That is what a contour on a map is, and it is how every performance surface in engineering is read. Curves crowded together mean the quantity changes fast in that region.

Limits come next and they are harder here than in one variable. In one variable there are two ways to approach a position. In the plane there are infinitely many routes. A limit exists only when every route agrees, so exhibiting two routes that disagree is the standard way to show a limit does not exist.

The partial derivative is the obvious next step and it is almost the right one. Hold every variable but one fixed and differentiate. That measures how the output responds to that variable alone.

Higher-order partials follow, and one convenience is worth naming. For the functions met in practice, differentiating with respect to one variable and then the other gives the same result in either order. This is a theorem with conditions rather than an identity, and it is worth knowing that it can fail.

Now the part that is genuinely subtle. Having both partial derivatives is weaker than being differentiable. A function can respond smoothly along each axis and still behave badly along a diagonal route. Differentiability means the surface is well approximated near the position by a flat plane, and that is the property everything later needs.

When it holds, the total differential says how the output moves when every input moves a little. Each partial derivative is multiplied by the change in its own variable, and the products are added. That single expression is the basis of every error-propagation calculation in the measurements course. Uncertainty in each input is carried through by its partial derivative, and the contributions combine.

The gradient collects the partial derivatives into one object with a direction. Two facts make it useful, and they are the reason it appears everywhere.

• The rate of change along any chosen direction is obtained from the gradient and that direction.
• Of all directions, the gradient itself is the one in which the function increases fastest.

That second fact is what every optimiser in the machine learning area relies on. Walking against the gradient is the fastest local decrease, which is the whole idea behind gradient descent. Here the same object tells you which way a loss surface climbs.

Optimisation then generalises cleanly. An interior maximum or minimum has a gradient of zero, because a non-zero gradient means there is a direction that improves the output. The second-order test says which kind of position it is. The new possibility is the saddle, which is a maximum along one direction and a minimum along another. Saddles have no analogue in one variable and they are common.

Constrained optimisation closes the topic. Often the variables are not free, because a fixed relation ties them together. A total power budget, a fixed board area or a required output level each does that.

One route is to solve the constraint for one variable and substitute. That works when the constraint is simple and fails when it is not. The Lagrange multiplier method avoids the substitution altogether. At the best position, the gradient of the objective lines up with the gradient of the constraint, and the multiplier is the factor relating them. The multiplier is not bookkeeping. It says how much the best value would improve if the constraint were relaxed slightly, which is often the number a designer wants.

What you should now be able to explain or do

• Read a level curve diagram and say where the quantity changes fastest.
• Show that a limit in the plane does not exist by finding two routes that disagree.
• Compute partial derivatives of any order and say what each measures.
• Explain why having partial derivatives is weaker than being differentiable.
• Write the total differential and use it to carry uncertainty through a calculation.
• Use the gradient to find the direction of steepest increase, and find a constrained optimum with a multiplier.

Check yourself