PE1-3.2 Statistical Decision Theory

What this is and why it exists

Something arrives, and it is noisy. You must decide which of two things sent it. That is the whole problem, and everything else in this subject is a special case of it.

There are three standard criteria for making that decision. They differ only in what you are allowed to assume.

The vocabulary

• Hypothesis — one of the possible truths you are deciding between.
• Prior probability — how likely each hypothesis was before you observed anything.
• Cost — the penalty attached to each kind of mistake.
• Bayes criterion — the rule that minimises the average cost.
• Likelihood ratio — the probability of the observation under one hypothesis, divided by its probability under the other.
• Threshold — the value the likelihood ratio is compared against.
• Minimax criterion — the rule that minimises the cost under the worst possible prior.
• Neyman-Pearson criterion — fixing one error rate and minimising the other.
• False alarm — deciding something is present when it is not.
• Missed detection — deciding nothing is present when something is.

The mental model

Every one of these criteria ends in the same shape of answer. Compute the likelihood ratio and compare it against a threshold. Only the threshold changes.

That is the single most useful sentence in this topic. Understand the likelihood ratio and the three criteria stop being three separate methods.

The ratio itself is intuitive. Ask how probable the thing you observed would be if the first hypothesis were true, and how probable it would be if the second were. If the first is much more probable, believe the first. The ratio measures how much more.

The Bayes criterion is the most general and needs the most. You must supply prior probabilities for the hypotheses and a cost for each kind of mistake. It then places the threshold to minimise the average cost, and it is optimal given those inputs. It generalises cleanly to more than two hypotheses. The observation space is divided into regions, one per hypothesis, and you choose the region the observation falls in.

The minimax criterion is for when the priors are unavailable. Instead of guessing, assume the prior that makes your performance worst, and place the threshold to minimise the cost under that assumption. It is the conservative choice. It costs performance whenever the worst case does not happen, which is most of the time, and it protects you when it does.

The Neyman-Pearson criterion is for when the costs are not comparable. This is common. In radar, what is the cost of missing an aircraft compared with the cost of a spurious blip? The two are not measured in the same units and no honest number exists. What you can specify is a tolerable false alarm rate. So fix the false alarm rate at an acceptable value and place the threshold to maximise the probability of detection subject to it.

That is why radar uses it. You can name a tolerable false alarm rate. You cannot name a sensible prior probability that an aircraft is present in a given cell.

Notice the pattern. Each criterion answers a question about what information you actually have. None of them is better than the others in the abstract.

What you should now be able to explain or do

Set up a binary hypothesis test and compute a likelihood ratio. Apply the Bayes criterion with given priors and costs, and extend it to more than two hypotheses. Say what minimax assumes and what it costs. Say what Neyman-Pearson fixes and why radar uses it. Explain why all three end in the same test with a different threshold.

Check yourself