EC-16.3 Responsible Data and Machine Learning, Privacy and Security

What this is and why it exists

Almost every device this area produces now collects data, and many contain a trained model.

The ethical questions therefore arrive with the hardware rather than with the research. They arrive for the engineer who ships the product rather than the one who built the model.

That is the perspective this topic takes, and it is the one the artificial intelligence elective does not cover.

The vocabulary

Informed consent — permission given knowingly, freely, and for a stated purpose.
Purpose limitation — using data only for what it was collected for.
Data minimisation — collecting only what is needed.
Retention — how long data is kept before deletion.
Distributional failure — a model performing worse for some groups than others.
Silent failure — a system giving a confident wrong answer with no indication.
Secure by design — security decided while designing rather than added afterwards.

The mental model

Collecting data about people carries three obligations, and they hold whatever a local regulation happens to say. They follow from the duty rather than from the law.

Consent that is informed and freely given. A stated purpose. And no quiet reuse for something else later. Reuse is the one most often violated, because data already collected feels free, and the person who consented consented to something narrower.

Two practices follow and both are cheap. Data not collected cannot leak. Data deleted on schedule cannot leak later.

Designing for the minimum needed, and setting a retention period at design time, is the cheapest protection available. It is also the one most often skipped. It is also the one that survives every subsequent failure of process, because there is nothing there to lose.

Trained models bring their own obligations to a product. Two failure shapes matter most.

A model that performs worse for some users than others ships that difference to everyone who uses the product. This is not exotic. It happens whenever the data a model learned from is not representative of its users, which is most of the time unless somebody checked.

A model that fails silently outside its training conditions is the second. It returns a confident answer that is wrong, with nothing to indicate the difference. Testing across the range of real users and real conditions is an engineering obligation. It is so in exactly the way testing a circuit at its temperature extremes is.

Explaining a decision is the next obligation. A person affected by an automatic decision is entitled to know why it was made.

If the system cannot produce a reason, that is a design limitation to be stated, not an inevitable property of the technology. It also affects where such a system may reasonably be used. A decision that cannot be explained should not be one with serious consequences for a person.

Security is a property of a design rather than a feature added to it. Default passwords, unsigned firmware updates and open debug ports on a shipped product are design decisions.

They are usually made by omission, which does not make them any less decisions. Security added after a product works is far weaker than security decided while it is being designed. The boundaries and the trust relationships are already fixed by then.

Finally, the life of a product after it ships. A device that cannot be updated cannot be fixed when a fault is found, and faults are found.

Two questions belong in the design phase. How will somebody report a vulnerability to you? And how will you get a fix onto devices already in the field? A product with no answer to either has decided, by default, that it will never be fixed.

What you should now be able to explain or do

State the three obligations attached to collecting data about people.
Apply data minimisation and a retention period as design decisions.
Recognise distributional and silent failure in a shipped model, and test for both.
Say what explaining a decision requires and where an unexplainable system should not be used.
Identify default passwords, unsigned updates and open debug ports as design decisions.
Plan for vulnerability reporting and field updates before shipping.

Check yourself

Which of the data obligations is violated most often?

Purpose limitation. Data already collected feels free to reuse, but the person consented to a narrower purpose than the one it is later put to.

Why is data minimisation the cheapest protection available?

Because data not collected cannot leak, whatever else fails. It survives every subsequent failure of process, since there is nothing there to lose.

What is a silent failure in a shipped model?

A confident wrong answer outside the conditions the model learned from, with nothing to indicate the difference. It is more dangerous than an obvious failure because nobody checks.

Why is security added later weaker than security designed in?

Because the boundaries and trust relationships are already fixed. Retrofitting works around a structure that was chosen without security in mind, rather than choosing a better structure.

Go deeper

We haven't checked most of these for screen reader use yet.

Back to Responsible Data and Machine Learning, Privacy and Security: work through the checklist