11.1 Signals and systems through an ML lens

What this is and why it exists

If you have studied signals and systems, you already know several of the central ideas in machine learning under different names. This topic makes the mapping explicit, because the transfer of intuition is the whole point and it works in both directions — your filtering instincts explain what a convolutional layer is doing, and the learned version tells you something about what a designed filter was choosing. Keeping the two worlds separate in your head is the only way to waste the advantage.

The vocabulary

• Convolution — sliding one function over another and integrating or summing the product.
• Cross-correlation — the same operation without one of them reversed.
• Impulse response — what a system outputs when given a single spike.
• Linear time-invariant — a system where scaling and adding inputs scales and adds outputs, and behaviour does not change over time.
• Sampling rate — how often a continuous signal is measured.
• Aliasing — high frequencies appearing as low ones because sampling was too slow.
• Frequency domain — a signal described by which frequencies it contains.
• Basis — a set of components any signal can be written as a combination of.

The mental model

The convolution in a network and the convolution in signal processing are the same operation with a sign convention between them. In signal processing, convolution reverses one of the two functions before sliding; the operation without that reversal is cross-correlation. What every deep learning framework calls convolution is, strictly, cross-correlation — and it does not matter, because the kernel's values are learned, so a reversed kernel is merely a different set of numbers, which training finds as readily.

That resolves the confusion, and then the useful part follows. A convolutional layer is a bank of learned finite impulse response filters. Each kernel is a filter; each output channel is that filter's response across the input. An edge-detecting kernel is a difference operator, which is a high-pass filter. A smoothing kernel is a moving average, which is a low-pass filter. A first layer that learns oriented edge detectors has learned a filter bank you could have designed — and that is exactly the connection the classical image topic made from the other side.

Which makes several network properties obvious rather than mysterious. Weight sharing is time or space invariance: the filter is the same wherever it is applied, which is precisely what makes a system time-invariant. Stride is decimation — take every second output — and it carries the same aliasing risk as decimating any signal, which is why smoothing before downsampling matters in both fields. Dilation is a sparse kernel, inserting zeros between taps, which widens the reach without adding coefficients. And the receptive field is the length of the cascaded impulse response: convolving filters in series produces a longer effective filter, and computing how long is the same arithmetic in both subjects.

Sampling theory governs data resolution choices, and the reasoning transfers directly. The sampling theorem says a signal is recoverable only if sampled at more than twice its highest frequency; below that, high frequencies fold down and appear as low ones that were never there — and the folded content is indistinguishable from real low-frequency content afterwards, which is why prevention rather than repair is the rule. The remedy is a low-pass filter before sampling, and the equivalent in image work is blurring before downscaling, which many resizing routines do for exactly this reason.

The transfer is that every resolution choice is a sampling decision. How often a sensor is read, how finely a time series is bucketed, at what resolution images are stored, how large the hop is between analysis windows: each sets a ceiling on the detail that can be represented, and detail lost there is not recoverable by any model. Choose the rate from the fastest thing you need to see, not from what is convenient — and remember the other side, since sampling far above what the phenomenon contains costs storage and computation for information that is not there.

Linear time-invariant systems and linear models share a structure, and it is the clearest bridge between the two subjects. An LTI system is completely characterised by its impulse response, and its output is the input convolved with it. A linear model is a weighted sum of inputs. Applied along a sequence, a linear model over a window of past values is an impulse response, and its coefficients are filter taps.

So the tools carry across. Superposition means analysing a complicated input as a sum of simple ones. The frequency response tells you which components a linear model amplifies and which it suppresses, which is a genuinely useful way to read a fitted time-series model. And stability is the same question: a recursive filter whose poles sit outside the unit circle diverges, and a recurrent model whose recurrence has the wrong scaling explodes — which is exactly the exploding gradient from the sequence topic, in the language you already had for it.

Fourier intuition is what makes feature design productive rather than mechanical. Any signal can be written as a sum of sinusoids, and looking at which frequencies it contains reveals structure the time-domain view hides. A vibration whose harmonics change describes a machine changing state, while its raw waveform looks like noise. A daily and weekly cycle in demand is two clear peaks in the frequency view and a mess in the time view. Speech is defined by its spectral shape, which is why every speech feature in the next topic is built in the frequency domain.

The practical rule that follows: when a model struggles on raw sequential data, look at its spectrum before reaching for a larger model. The structure the model is failing to find is frequently plain in the frequency view, and giving it spectral features is cheaper and more reliable than asking it to discover the transform for itself.

And the bridge is two-way, which is worth stating deliberately. From signals into machine learning: filtering, sampling, stability and spectral thinking all apply and are all under-used by people who did not study them. From machine learning back into signals: a learned filter bank finds the filters that matter for a task rather than the ones a designer expected, and that is genuinely new information about the problem. Read a trained first layer as a filter bank and ask what it chose — it will occasionally tell you something about your data that no amount of designing would have.

What you should now be able to explain or do

State the relationship between convolution and cross-correlation and say why the difference does not matter for learned kernels. Read a convolutional layer as a bank of learned filters, and map weight sharing, stride, dilation and receptive field to their signal-processing counterparts. Apply the sampling theorem to any resolution choice, in both directions. Explain why a linear model over a window is an impulse response, and connect stability to exploding gradients. Use the spectrum to diagnose a model struggling on sequential data. Read a trained first layer as a filter bank.

Check yourself