6.11 Image processing fundamentals for machine learning

What this is and why it exists

Before convolutional networks there was a whole craft of filters, edges and hand-designed descriptors, and it did not stop being useful when networks arrived. It is what you fall back on with two hundred images and no accelerator, it is what runs on a microcontroller, and it is what makes the first layers of a convolutional network legible — because those layers learn very nearly the filters this topic asks you to apply by hand. This is also the honest baseline that everything later in the module has to beat.

The vocabulary

• Channel — one of the numbers stored per pixel, such as red, green or blue.
• Colour space — the meaning assigned to those numbers.
• Kernel — a small array of weights slid across an image.
• Convolution — the sliding-and-summing operation itself.
• Edge — a place where intensity changes sharply.
• Threshold — the cut that turns a greyscale image into a two-valued one.
• Morphology — operations that grow, shrink and clean up shapes.
• Descriptor — a fixed-length summary of an image or a region, usable as features.

The mental model

An image is an array, and getting the conventions right removes most early confusion. The scikit-image guide states them exactly: "Images in scikit-image are represented by NumPy ndarrays", two-dimensional greyscale images are "indexed by rows and columns (abbreviated to either (row, col) or (r, c)), with the lowest element (0, 0) at the top-left corner", and this is distinguished from Cartesian coordinates "where x is the horizontal coordinate, y - the vertical one, and the origin is at the bottom left". A colour image "is a NumPy array with an additional trailing dimension for the channels". Rows before columns, origin at the top left, channels last — three facts that between them account for a great many transposed and upside-down images. Note also that deep learning frameworks commonly put channels first instead, so a conversion sits at the boundary between these libraries.

The colour space decides which features are visible. The default arrangement mixes brightness into all three channels, so a shadow changes every one of them and colour-based rules become fragile. Converting to a space that separates a brightness value from the colour itself lets you ask about colour without asking about lighting, which is why colour-based selection is usually done there. Converting to greyscale discards colour entirely, and for shape and texture work that is a feature rather than a loss: one channel instead of three, a third of the computation, and nothing important lost.

Filtering is convolution with a small kernel, and three families cover most needs. Smoothing kernels average a neighbourhood, which suppresses noise and blurs detail — Gaussian smoothing is the standard, and its width is the setting that trades one against the other. Sharpening does the reverse, amplifying local differences. Derivative kernels respond to change: one pair estimates the rate of change horizontally and vertically, and combining them gives an edge strength and a direction at every pixel.

Edge detection builds on those. The classic multi-stage detector — available as canny, documented as "edge filter an image using the Canny algorithm" — smooths first, computes gradients, keeps only local maxima along the gradient direction so edges come out one pixel wide, and then uses two thresholds so a weak edge survives if it connects to a strong one. Understanding those stages matters because its two settings behave differently: the smoothing width decides which scale of detail counts as an edge, and the thresholds decide how much survives.

Thresholding and morphology are what turn an edge image into objects. Thresholding converts to two values, either with one cut for the whole image or with a cut computed per region when lighting is uneven. Morphological operations then clean up the result using a small shape: erosion shrinks bright regions and removes specks, dilation grows them and closes small gaps, opening is erosion then dilation and removes noise while preserving size, closing is the reverse and fills holes. Opening then closing, with a shape the size of the noise you want gone, fixes an enormous proportion of real segmentation problems and costs nothing.

Classical descriptors are the step from pixels to features. Three worth knowing, all present in scikit-image's feature module. The histogram of oriented gradients — hog, "extract Histogram of Oriented Gradients (HOG) for a given image" — divides the image into cells, builds a histogram of edge directions in each, and normalises across blocks of cells; it describes shape while tolerating small shifts and lighting changes, and it was the backbone of pedestrian detection for years. Local binary patterns — local_binary_pattern, "compute the local binary patterns (LBP) of an image" — compare each pixel with its neighbours and encode the comparison as a small number, then histogram those; it is a compact and effective texture descriptor. And corner responses such as corner_harris, "compute Harris corner measure response image", find points that are distinctive in every direction, which is what makes them repeatable across views and therefore useful for matching.

When do these still beat learned features? When you have a few hundred labelled images rather than a few thousand. When the task is narrow and well defined — reading a gauge, checking a part is present, counting objects on a plain background — and a designed feature captures the whole problem. When the compute budget is a small embedded processor. When you need to explain the decision, because a threshold on an edge count is explicable in a way an activation is not. And when you need the result this week and cannot label a dataset first. The mistake is not choosing the classical route; it is choosing the learned route without ever measuring the classical one.

So build the baseline. Take a small labelled image set, convert to greyscale, extract a descriptor for each image, and put the resulting feature vectors into any classifier from the classical module. It runs in seconds on a laptop, and it produces the number every later approach must beat. Frequently it is better than expected, and occasionally it is enough.

What you should now be able to explain or do

State the array conventions for images and say where a channels-first conversion is needed. Say what a colour space change buys and when greyscale is the right choice. Apply smoothing, sharpening and derivative filters, and describe what each does. Explain the stages of the multi-stage edge detector and what its two settings control. Threshold and clean up with erosion, dilation, opening and closing, choosing the structuring size deliberately. Extract a shape, a texture and a corner descriptor and say what each captures. State the conditions under which classical features still win. Build and report a classical baseline for a small image set.

Check yourself