13.2 Capstone 2 — vision or audio system

What this is and why it exists

The second capstone is a perception system — vision or audio — that somebody can try in a browser. Transfer learning makes a strong result possible on a few hundred examples, and collecting some of your own data teaches things no downloaded set can. The rule that decides whether this project lands is at the end and it is blunt: stop at the notebook and it reads as unfinished. An interactive demonstration is what a non-technical reviewer experiences, and no demonstration means no evidence.

The vocabulary

• Curation — assembling and cleaning a dataset rather than downloading one.
• Class imbalance — some categories far rarer than others, which yours will be.
• Transfer learning — starting from pretrained weights.
• Augmentation — manufacturing variety from the examples you have.
• Latency budget — how long a response may take to feel interactive.
• Interactive demonstration — a hosted page where somebody supplies input and sees output.

The mental model

Collect some of your own data, and this is the recommendation that makes the project yours. Two hundred photographs you took, or an hour of audio you recorded, teaches more about data than any public set: you meet the ambiguity of your own labels, the fact that half your images were taken in the same lighting, the class you thought was one thing and is three, and the examples that are genuinely undecidable.

It also makes the project distinctively yours, which matters when reviewers have seen the same public dataset a hundred times. Choose something you can actually photograph or record — objects around you, sounds in your environment, a task from work or from a hobby — and be honest in the write-up about where the data came from and what it therefore does not cover.

Two obligations come with collecting. If people appear in it, you need their consent, and their data is subject to the obligations from the production module. And record the collection conditions — where, when, with what device — because those explain your results and are exactly what a reviewer will ask about.

Curation is most of the work and it is worth doing visibly. Define your classes precisely, write the definition down, and label a hundred examples yourself before deciding it is right — the definition will change, and changing it after a thousand labels is the expensive version. Split by capture session, not at random, for the reason the sensor topic gave: photographs from one session or recordings from one visit resemble each other far more than they resemble anything else, and a random split scores your model on data it effectively saw. Expect imbalance, because real collection produces it, and handle it with the metric and threshold first as the imbalance topic said.

Transfer learning and augmentation carry the whole project, and they are what make a few hundred examples enough.

Start from pretrained weights and follow the sequence exactly: freeze the backbone, train the head until it is doing something, then unfreeze progressively at a much smaller learning rate. Reversing that order scrambles the features before the head has learned anything. Use the same preprocessing the weights were trained with, since a mismatch quietly costs several points and reports nothing.

Augment in ways the problem allows — the rule from the regularisation topic, and its consequences differ by domain. For images: crops, flips where the label survives them, rotations, colour and brightness variation, and mild blur. For audio: time shifts, added recorded background noise, small pitch and speed changes, simulated room response, and masking spans of time and frequency. Augmentation is usually the largest single improvement available on a small collection, and it is cheaper than collecting more.

Latency and size are what turn a notebook result into a system, and this constraint is the point of including them.

Set the budget first: for an interactive demonstration, a response within about a second feels immediate and several seconds feels broken. Then work to it with the tools from the optimisation topics — a smaller architecture where one suffices, quantisation, an exported portable model, and a check that the preprocessing is not the slow part, which it frequently is. Measure end to end on the hardware you will actually deploy on, including the upload, because a benchmark of the model alone is not the number a user experiences.

Size matters for a different reason: free hosting has memory limits, and a model that will not fit does not run at all. Knowing the limit before choosing the architecture is the same rule the embedded topic gave.

Then the demonstration, which is the single most effective portfolio item in this module.

Build it with one of the small Python interface libraries — either the one designed for machine-learning demonstrations or the general data-application one — both of which turn a function into a web interface in a few dozen lines. Host it publicly on a free hosting service for such applications. A reviewer clicks a link, uploads something of their own, and sees your system work. That experience is worth more than a page of metrics, and it is what a non-technical hiring manager can actually assess.

Four things make a demonstration good rather than merely present. Example inputs provided on the page, so somebody can try it in one click without finding a file. The confidence shown alongside the prediction, and a stated behaviour when confidence is low, because a system that says "I am not sure" is more impressive than one that always answers. An honest note about what it handles and what it does not — the classes it knows, the conditions it was trained on — displayed on the page rather than buried. And a warm start or a clear message while loading, since free hosting sleeps and a reviewer meeting a thirty-second blank page concludes it is broken.

And the closing point, stated plainly because it is the difference between this project counting and not. The model is the part you enjoyed; the demonstration is the part anybody else can see. A notebook with excellent results and no running system is, to a reviewer, indistinguishable from a tutorial that was followed — and "no demo" reads as "not finished", whatever the notebook contains.

What you should now be able to explain or do

Collect some of your own data and say what that teaches, with the consent and record-keeping obligations. Define classes precisely, label a sample before committing, and split by capture session. Apply transfer learning in the right order with matched preprocessing. Choose augmentations the problem allows, for images or audio. Set a latency budget and meet it, measuring end to end on the deployment hardware. Fit the hosting memory limit. Build and host an interactive demonstration with examples, confidence, honest scope and a loading message.

Check yourself