11.4 Audio and speech machine learning

What this is and why it exists

With the previous two topics behind you, audio machine learning has no front-end mystery: you know what the features are and why they are built that way. This topic is what gets built on top — classifying sounds, spotting a keyword, transcribing speech, telling speakers apart, producing speech — and the failure that separates a demonstration from a product, which is that real audio is noisy, accented, distant and compressed, and a pipeline tuned on clean recordings degrades sharply the moment it leaves the room.

The vocabulary

• Audio classification — labelling a whole clip.
• Event detection — finding where within a recording something occurs.
• Keyword spotting — continuously listening for a small set of phrases.
• False accept and false reject — waking when it should not, and failing to wake.
• Streaming transcription — producing text as the person speaks.
• Diarisation — deciding who spoke when.
• Far-field — recorded at a distance, with room reverberation.
• Codec artefact — distortion introduced by compression for transmission.

The mental model

Classification is the entry point and it reuses the vision module wholesale. Compute a mel spectrogram, treat it as an image, and train a convolutional network on it. Augmentation transfers with one adjustment: augment in a way the physics allows. Time shifting, adding recorded background noise, small pitch and speed changes, and simulated room reverberation are all realistic. Masking spans of time and bands of frequency in the spectrogram is the standard augmentation here and works well. Flipping the time axis is not realistic for speech, and flipping the frequency axis is not realistic for anything.

Event detection is classification with a location, and its complications are the interesting part: events overlap, boundaries are genuinely ambiguous, and the labels are usually weak — somebody marked that a clip contains a dog, not the second it barked. Evaluate at the event level with a tolerance for boundary error, not frame by frame, for the reason the sequence-labelling topic gave: frame accuracy can look excellent while every event is misplaced.

Keyword spotting is a different engineering problem from everything else here, because it runs continuously on a device, on battery, and must almost never wake by mistake. The consequences shape the design entirely. The model must be tiny, which is the next-but-one topic. It usually runs in stages: a very cheap always-on detector, then a larger confirming model, then possibly a server — so the expensive stages only run when the cheap one fires. And the operating point is chosen from the cost of each error: a false accept is a device waking in the middle of a conversation, a false reject is a person repeating themselves. Both are annoying, they are not equally annoying, and the threshold is a product decision made with real numbers rather than a default.

Transcription splits into batch and streaming, and the streaming case is substantially harder. A batch system has the whole recording, so it can use context from after a word to decide what it was — which is exactly the bidirectional advantage from the language module, and it is worth a great deal.

A streaming system cannot. It must emit text as it goes, so each decision is made from the past and a small lookahead, and it must handle revising what it already showed as more arrives. Latency and accuracy trade directly: more lookahead is more accuracy and more delay. Design that trade explicitly — a caption may tolerate a second, a voice interface may not — and note the two-system pattern that works well, where a fast streaming model gives immediate text and a batch model corrects the transcript afterwards.

Practically, open-weight transcription models are now strong, multilingual and straightforward to run locally, which changed this from a project into an afternoon. What has not changed is the evaluation, which is where the next section is.

Speaker identification and diarisation are needed by any system handling meetings or calls. Identification uses an embedding: a model maps a stretch of speech to a vector where the same speaker lands close, exactly the representation idea from the deep learning module, and comparison is then a distance. Diarisation answers who spoke when — segment the audio, embed each segment, cluster the embeddings, and assign labels.

Its hard cases are worth knowing because they are common rather than exotic: overlapping speech, where two people talk at once and simple clustering has no answer; short turns, where a one-word interjection gives too little audio for a reliable embedding; similar voices; and not knowing how many speakers there are, which most clustering needs told or must estimate. Meeting recordings contain all four in the first minute.

Speech synthesis and voice interfaces close the topic, and latency is what decides whether it feels like a conversation. Quality is now high enough that naturalness is rarely the constraint; the design question is the loop. A person speaks, the system detects the end of speech, transcribes, decides, synthesises, and plays — and the sum of those must be about the pause a person would leave. Overlapping the stages is what makes that possible, as the multimodal topic described, and stopping playback the instant the person starts speaking is not a refinement; without it the interface feels broken, because people interrupt constantly.

Two obligations belong with synthesis and they are part of the engineering. A synthetic voice should be identifiable as synthetic wherever a listener might reasonably assume otherwise. And cloning a specific person's voice requires that person's consent, without exception.

Then the trap this topic exists for: testing only on clean audio. Studio recordings, one speaker, close microphone, no background — a pipeline tuned there degrades sharply in the field, and the degradation is not gradual.

Four things real deployments meet. Noise: traffic, fans, other people, music. Accents and code-switching, which in India is the normal case rather than an edge case — speakers move between languages within a sentence, and a system evaluated on one language handles that badly. Far-field capture: a microphone across the room adds reverberation and drops the level, and reverberation smears exactly the time structure the features encode. Codec artefacts: audio over a telephone or a conferencing system has been compressed, band-limited and sometimes had silence suppressed — and band-limiting removes the high frequencies where fricatives live, which is why telephone audio confuses similar consonants.

So build the evaluation set from real conditions before tuning anything. Recordings from the actual microphones, in the actual rooms, from the actual speakers, over the actual transmission path. Report the error rate per condition rather than overall, because an average across clean and difficult audio hides both. And where you cannot collect enough real difficult audio, simulate it — add measured noise at known levels, convolve with recorded room responses, pass audio through the codec you will actually use — which is cheap and is far closer to reality than testing on studio recordings.

What you should now be able to explain or do

Build an audio classifier from mel spectrograms and choose augmentations the physics allows. Evaluate event detection at event level with boundary tolerance. Design keyword spotting as staged detection and choose the operating point from the cost of each error. State why streaming transcription is harder and design the latency-accuracy trade, including the two-system pattern. Build identification from embeddings and diarisation from clustering, and name the four hard cases. Design a voice loop around latency and interruption, with the disclosure and consent obligations. Build an evaluation set from real conditions and report per condition, simulating what you cannot collect.

Check yourself