4.13 Bayesian networks: representation and inference

What this is and why it exists

A Bayesian network is a picture of conditional independence. The graph is the substantive assertion — which things directly influence which — and the tables are the detail. Its value is compactness: a joint distribution that would need thousands of numbers fits in a handful, because the graph says most of those numbers were never needed. This topic is how to read one, how to build one, how to answer a question with it exactly, and when a probabilistic representation is the right choice over a symbolic one.

The vocabulary

• Random variable — a quantity that can take one of several values with probabilities.
• Joint distribution — the probability of every combination of values of every variable.
• Conditional probability — the probability of one thing given that another is known.
• Conditional independence — two variables becoming independent once a third is known.
• Parent — a node with an arrow into another; child — the node it points at.
• Conditional probability table — the probabilities of a variable's values given each combination of its parents' values.
• Evidence — the variables whose values you have observed.
• Query — the variable whose probability you want, given the evidence.

The mental model

Begin with the problem the network solves. A joint distribution over ten yes-or-no variables has over a thousand entries; over twenty, more than a million. Nobody can supply those numbers, and nobody could check them. Yet with the joint distribution you could answer any question, so the goal is to keep that power while storing far less.

Conditional independence is what makes that possible. Two variables are conditionally independent given a third when, once you know the third, learning one tells you nothing more about the other. Rain influences whether the grass is wet, and whether the grass is wet influences whether it is slippery — but once you know the grass is wet, learning that it rained tells you nothing further about slipperiness, because rain only ever acted through the wetness. The information flowed through, and knowing the intermediate blocks it.

A Bayesian network encodes exactly such statements as a graph. An arrow means direct influence. The absence of an arrow is the assertion that matters: it says any influence between those variables is mediated by others in the graph. The graph is a compact statement of assumptions, not a picture of the data, and reading it as a summary of correlations is the most common misreading.

The payoff is that each variable needs probabilities only for its own values given its parents' values, rather than given everything. Ten variables each with at most two parents need a few dozen numbers instead of a thousand, and those numbers are ones a domain expert can actually supply, because each asks a local question: given these two things, how likely is this one?

Building a network from a description, and the modelling decisions in it.

Name the variables and their possible values first, and keep them few — every extra value multiplies a table. Then decide the arrows. The reliable method is to add variables in causal order, causes before effects, and for each new variable ask which of the ones already added directly influence it. Arrow direction is a modelling decision with real consequences: causal direction usually gives sparser graphs and tables that are easier to fill in, because "how likely is this symptom given this disease" is a question a clinician can answer, while the reverse question depends on how common the disease is in the population and changes when that changes. A network built against the causal direction is not wrong — it can represent the same distribution — but it will need more arrows and its numbers will be harder to obtain and less stable.

Then fill the conditional probability tables. Each row corresponds to one combination of the parents' values, and the entries in that row are the probabilities of the child's values, which must sum to one across the row. A variable with no parents gets a simple prior.

Reading a table correctly is where the most common error in this topic lives. Every number in it is conditioned on the parents, so it answers "given the parents are like this, how likely is the child like that" — and never the reverse. Confusing the probability of the evidence given the hypothesis with the probability of the hypothesis given the evidence is the classic mistake, and it is not a small one: a test that is positive in 99 cases out of 100 when the disease is present tells you very little about whether a positive result means disease, if the disease is rare. The direction of conditioning is the whole content of the number, so say it out loud when reading a row.

Exact inference answers a query by summing the joint probability over every value of the variables you neither observed nor asked about. Done directly, this enumerates combinations and is exponential. Done sensibly, you notice that the same partial sums appear repeatedly and compute each one once, working from the leaves inward — the same reuse-of-partial-results idea that makes the sequence algorithms in the next topic efficient.

That improvement is large and it does not change the ending. Exact inference becomes infeasible as networks grow, and the property that decides it is not the number of nodes but how densely connected the graph is — a large tree-like network is straightforward, and a modest densely connected one is not. When it stops being feasible, the standard answer is sampling: generate many complete assignments consistent with the network, keep those matching the evidence, and estimate the answer from the proportions. That trades an exact answer for one with a known accuracy that improves with the number of samples, which is usually the right trade and is the reason approximate methods exist.

Then the judgement this topic is really for: when is a probabilistic representation the right one?

When evidence is uncertain, partial or conflicting. A logical representation must treat a statement as true or false; two symptoms pointing at different diagnoses give a contradiction, and a contradiction in classical logic lets anything be derived. A probabilistic one weighs them and returns a distribution, which is what the situation actually calls for.

When you need to reason in both directions. The network is built causally and queried either way: given the disease, how likely is the symptom, and given the symptom, how likely is the disease. That reversal is exactly what diagnosis needs and what a rule base written in one direction cannot do.

When you need to say how sure the system is. A probability is a statement about confidence that downstream decisions can act on, including the decision to ask for more evidence. A symbolic conclusion is bare.

And when the answer should change as evidence arrives, without rewriting rules — a new observation is new evidence, not a new rule.

The costs are real and worth stating. You need numbers, and getting them is the acquisitional problem from the representation topic in a new form — from data if you have it, from experts if you do not, and expert-supplied probabilities are known to be poorly calibrated. Inference costs more than lookup. And the output is a distribution, which is more informative and harder to explain to somebody expecting an answer. Choose it when uncertainty is central to the problem rather than incidental, and choose a symbolic representation when the domain really is definite and what you need is an explanation of the reasoning.

What you should now be able to explain or do

State the problem a network solves and why the joint distribution is unusable directly. Define conditional independence and give the blocking example. Say what an arrow means and what the absence of one asserts. Build a network in causal order and justify arrow direction by which questions are answerable and stable. Fill and read a conditional probability table, saying the direction of conditioning aloud. Answer a query by exact inference and explain the reuse that makes it tractable. Say what property decides when exact inference stops being feasible and what replaces it. Argue when a probabilistic representation beats a symbolic one, and name its costs.

Check yourself