11.2 DSP features for machine learning

What this is and why it exists

A raw waveform is a long sequence of numbers with almost none of its structure visible. Spectral features make that structure explicit, and they are the front end of every audio and vibration system in production. The temptation is to skip them, on the grounds that deep learning learns its own features — and on realistic data budgets that is a losing trade, because the transform you would be asking the network to discover is one you can compute exactly, for free, from a hundred years of theory.

The vocabulary

• FFT — an efficient algorithm for the discrete Fourier transform.
• STFT — the transform applied to short overlapping windows, so the result varies with time.
• Window function — a shaping function applied to each segment before transforming.
• Hop length — how far the window advances between frames.
• Spectrogram — the magnitudes of a short-time transform, as an image of time against frequency.
• Mel scale — a frequency warping that spaces bands as hearing does.
• MFCC — a compact summary of a mel spectrogram.
• Spectral leakage — energy spreading across bins because a segment does not contain whole cycles.

The mental model

One transform underlies everything here. The discrete Fourier transform expresses a segment as a sum of sinusoids and returns how much of each frequency is present; the fast algorithm makes it cheap enough to run constantly. Applied to a whole recording it tells you what frequencies are present overall and nothing about when — which is useless for anything that changes, and everything interesting changes.

So apply it to short windows, advancing across the signal: that is the short-time transform, and its magnitudes displayed as time against frequency are a spectrogram. That one representation is the foundation of nearly all audio machine learning, and it also converts an audio problem into an image problem, so every technique from the vision module becomes available.

Three settings decide what the model sees, and they are the real content of this topic.

Window length sets the trade between time and frequency resolution, and the trade is unavoidable rather than a limitation of the algorithm: a short window locates events precisely in time and resolves frequencies poorly, a long window does the reverse. Choose from the phenomenon — short windows for percussive events and machine faults, longer ones for pitch and harmonic structure.

Hop length decides how much frames overlap. Smaller hops give smoother output and more frames to process; a hop of about a quarter of the window is a common starting point.

Window shape exists because of spectral leakage. A segment cut abruptly out of a signal contains partial cycles, which the transform represents by spreading energy across many bins — so a pure tone appears as a smear, and small components near a large one disappear underneath it. Tapering the segment smoothly to zero at both ends reduces that spreading considerably. A rectangular window — meaning no window at all — is a choice, and usually the wrong one, and the confusing spectra people show around this are almost always leakage.

Mel features and their compact form exist because human hearing is not linear in frequency. We discriminate finely at low frequencies and coarsely at high ones, so a linear frequency axis spends most of its resolution where the ear spends least attention. A mel spectrogram warps the axis to match, and groups the fine bins into a smaller number of bands — commonly a hundred or so — which loses nothing perceptually and reduces the size substantially.

The coefficient form goes further: apply a cosine transform to the log of the mel spectrum and keep the first coefficients. The effect is worth understanding rather than accepting. The log turns the source-and-filter product from the next topic into a sum, and the cosine transform separates the slowly varying part — the vocal tract's shape — from the rapidly varying part — the pitch. Keeping the first coefficients keeps the shape and discards the pitch, which is exactly right when the question is what was said and exactly wrong when it is who said it. These remain the standard features for speech, and knowing what they threw away tells you when to use something else.

The libraries implement all of it, and knowing which to reach for saves reimplementing standard transforms. For general signal work, scipy.signal has the pieces: ShortTimeFFT for a parametrised short-time transform with its inverse, spectrogram for a spectrogram from consecutive transforms, butter for "Butterworth digital and analog filter design", filtfilt to "apply a digital filter forward and backward to a signal", resample and resample_poly for rate conversion, and get_window as a "convenience function for creating various windows".

For audio specifically, librosa.feature.melspectrogram and librosa.feature.mfcc produce the two representations above, and the coefficient function takes either the audio directly or a precomputed mel spectrogram through its S argument — which matters, because computing the mel spectrogram once and deriving several things from it is both faster and clearer than recomputing.

Filtering and denoising come before any of it, and they affect results more than the model choice does. Restrict to the band the phenomenon occupies and everything outside is noise you no longer have to be robust to. Remove a constant offset, which otherwise dominates the lowest bin. Filter out mains interference, which is a strong narrow line in every recording made near a supply. And filter with zero phase where timing matters — an ordinary filter delays the signal by a frequency-dependent amount, and running it forwards and backwards cancels that, which is what filtfilt is for.

Then the trap, stated with its condition. Networks can learn from raw waveforms, and at very large data scales they do so successfully. On realistic data budgets — thousands of examples rather than millions — spectral features still win, and by a wide margin. The transform is not something the model needs to discover; it is exact, cheap, well understood, and it removes an enormous amount of learning the network would otherwise spend capacity on.

The strong middle position, worth knowing because it is what most production systems do: compute a mel spectrogram and train a convolutional network on it as an image. The signal processing supplies the representation, the network learns the task-specific part, and the whole vision toolkit applies. And treat the windowing settings as hyperparameters rather than defaults, because they materially change what the model sees — a model that fails at one window length and succeeds at another is a common and entirely avoidable surprise.

What you should now be able to explain or do

Explain why a whole-signal transform is useless for changing signals and what the short-time version fixes. Choose window length, hop and shape deliberately, and state the time-frequency trade as unavoidable. Recognise spectral leakage and know why a rectangular window is usually wrong. Say what the mel warping does and what the coefficient form discards. Use the library functions for transforms, filter design, zero-phase filtering, resampling and windows. Filter, remove offsets and suppress interference before feature extraction. State the condition under which spectral features beat raw waveforms, and build the mel-spectrogram-plus-network arrangement.

Check yourself