PE1-3.1 Discrete-Time Random Processes

What this is and why it exists

Most real signals are random, and most random signals are not white. Successive samples are related to each other.

This topic gives three ways to describe that relationship with a handful of numbers. Almost all of statistical signal processing assumes one of the three.

The vocabulary

• Discrete-time random process — a sequence of random values, one per sample instant.
• White noise — a sequence whose samples are uncorrelated with each other.
• Autocorrelation — how strongly a sequence resembles a shifted copy of itself.
• Autoregressive process — each sample is a weighted sum of previous samples, plus white noise.
• Moving average process — each sample is a weighted sum of previous noise inputs.
• ARMA process — a model with both kinds of term.
• Yule-Walker equations — the linear system relating autoregressive parameters to autocorrelation.
• Model order — how many past terms the model uses.

The mental model

Think of all three models as one machine. White noise goes in at one end, a filter shapes it, and a correlated sequence comes out. The three models differ only in the shape of that filter.

An autoregressive model has feedback. Each new sample is a weighted sum of the samples that came before it, plus a fresh white input. Because the output depends on its own past, its influence dies away gradually and never quite ends.

A moving average model has no feedback. Each sample is a weighted sum of the last few white inputs. Its memory is exactly as long as the number of terms and then stops.

An ARMA model has both. It fits a wider range of shapes with fewer parameters, which is its whole attraction.

In practice the autoregressive model dominates, and the reason is not that it fits better. It is that estimating it is a linear problem. Write down the relation between the model parameters and the measured autocorrelation and you get linear equations. They are the Yule-Walker equations, and any computer solves them directly. Moving average estimation is genuinely harder, because the relationship between its parameters and the autocorrelation is not linear. ARMA inherits that difficulty from its moving average half.

That is the useful thing to carry away. Choosing a model is not only about which describes the data best. It is also about which one you can actually fit.

Model order is a real decision with a penalty on each side. Too few terms and the model misses structure that is really there, so the residual is not white and information is left unused. Too many terms and the model starts describing the noise in your particular recording, which will not repeat. That second failure is the more dangerous one, because it looks like a better fit on the data you have.

The honest test is whether what is left over looks white. If the residual still has structure, the model is too small. If the fit improves only slightly for each extra parameter, the extra parameters are describing noise.

What you should now be able to explain or do

Describe all three models as one filter driven by white noise. Say what distinguishes autoregressive from moving average memory. Explain why the autoregressive model dominates in practice, and name the equations behind that. Say what goes wrong with too few and with too many parameters. Use the whiteness of the residual as a check on model order.

Check yourself