8.14 Multimodal models

What this is and why it exists

Images, audio and documents are now first-class input rather than a separate field, and that opens a large class of features: reading a photograph, transcribing a call, extracting structure from a scanned invoice. Real products are almost always pipelines of several components rather than one model doing everything. The trap is skipping modality-specific evaluation, because a model that describes photographs beautifully can misread a table, a chart or handwriting, and only a targeted test tells you which.

The vocabulary

• Vision-language model — one taking images alongside text and producing text.
• Grounding (in vision) — pointing at where in an image something is.
• Speech recognition — turning speech into text; speech synthesis — the reverse.
• Diarisation — deciding who spoke when.
• Word error rate — transcription accuracy, counting insertions, deletions and substitutions.
• Optical character recognition — reading characters from an image.
• Layout analysis — recovering the structure that gives those characters meaning.
• Barge-in — a person interrupting a spoken response.

The mental model

Vision-language understanding is the most mature of these capabilities, and the honest account of it is uneven. Describing a scene, answering a question about a photograph, reading a sign, judging what is happening: reliable. Reading a dense table without transposing a row, tracing a value from a chart, interpreting handwriting, counting more than a few objects, judging fine spatial relationships, reading small text at the edge of an image: much less so, and confidently wrong when wrong.

The practical consequence is that "can it read images" is the wrong question, and "can it read my images" is the right one. Fifty of your own examples, with the correct answers written down, will tell you in an afternoon what no general assertion can — and that same set becomes the regression suite when a model or a pipeline stage changes.

Three habits improve results markedly. Give the resolution the detail needs, since fine print and small chart labels are lost when an image is downscaled, and cropping to the relevant region beats sending a whole page. Ask for structure rather than prose — a schema with fields is checkable, a paragraph describing a table is not. And require the model to say when it cannot read something, for the same reason as everywhere else: the alternative is a plausible number.

Speech splits into two directions with one hard constraint. Recognition turns audio into text, and modern systems are strong on clear speech in well-represented languages and materially worse on accents, code-switching, background noise, overlapping speakers, and technical vocabulary — which is to say, worse on real recordings than on the audio anyone demonstrates. Measure with word error rate on your own audio, and look at which words are wrong: names, numbers and domain terms carry most of the meaning, so an error rate that looks acceptable can still be useless if the errors sit there. Supply a vocabulary hint where the interface allows one; it is the cheapest large improvement available.

Synthesis is the reverse and its quality is now high enough that the interesting decisions are about disclosure and consent rather than naturalness. Two things follow and both are the engineering rather than an appendix: a synthetic voice should be identifiable as synthetic where a listener might reasonably assume otherwise, and cloning a specific person's voice requires that person's consent, without exception.

Realtime voice is where latency is the design, and it is the constraint that shapes everything. A conversation tolerates roughly the pause a person would leave, and every stage spends part of it: detecting the end of speech, transcribing, generating, synthesising, playing. Stack them naively and the gap is uncomfortable, so real systems overlap the stages — transcribe as the person speaks, begin generating on a partial transcript, start synthesising the first phrase before the last is written. They also handle barge-in, stopping playback the moment the person starts speaking again, which people do constantly and which feels broken when it is missing. Both are the difference between a demonstration and something usable.

Document extraction is one of the highest-value applications in business settings, and it is where the pipeline shape is clearest. Reading characters is not the hard part; recovering the layout is. A table is meaningful because of which cell sits under which header; a form is meaningful because of which label a value belongs to; a multi-column page read left to right is nonsense. So the useful pipeline is: character recognition, layout analysis, then a model that reads text with its positions and produces a validated structure.

Three notes worth carrying. Digital documents contain their text already, so extracting it directly is exact and free, and running recognition over a page that was never scanned is a self-inflicted error rate. Validate what comes out against what you know — totals that add up, dates in range, identifiers matching a format — because a wrong digit in an amount is worse than a refusal. And route by confidence: high-confidence extractions pass, low-confidence ones go to a person. That last one is what makes these systems work in practice, and it needs a confidence signal designed in from the start rather than inferred afterwards.

Video, briefly, because it is the least settled. Understanding is mostly frames plus audio at present, with the genuinely hard part being events that only exist over time — a gesture, a sequence, a change. Generation is advancing quickly and is expensive, hard to control precisely, and moving too fast for anything specific written today to be worth relying on. What is worth learning now is the shape of the problem — that time is the dimension the current approaches handle least well — rather than any particular system.

And the closing point, which applies to all of it: build pipelines, not monoliths. A system that transcribes with a dedicated recogniser, extracts with a document pipeline, and reasons with a language model over the results is more accurate, cheaper, and debuggable stage by stage. One model asked to do everything gives you one number when it fails and nowhere to look.

What you should now be able to explain or do

Give an honest account of what vision-language models do well and badly, and test on your own images. Use resolution, cropping and structured output to improve results. Measure transcription with word error rate and by which words are wrong, and use vocabulary hints. State the disclosure and consent requirements for synthesis. Explain why realtime voice overlaps its stages and what barge-in is. Design a document pipeline with layout analysis, validation and confidence routing, and extract digital text directly. Say what makes video hardest and why specifics are not worth memorising. Argue for pipelines over monoliths.

Check yourself