4.4 Task environments and the four agent types

What this is and why it exists

The six properties of a task environment are not labels to memorise for an examination. They decide the architecture. A partially observable, stochastic, sequential environment rules out the simplest agent structure before you have written a line of code, and knowing that saves you from discovering it three weeks in. Classify the environment first, and the agent type is largely a consequence.

The vocabulary

• Fully observable — the sensors give the complete state relevant to the decision.
• Deterministic — the next state is fixed by the current state and the action.
• Episodic — each decision stands alone, with no consequences for the next.
• Static — the world does not change while the agent is deciding.
• Discrete — states, time and actions come in distinct steps.
• Multi-agent — other agents are present whose behaviour must be considered.
• Internal state — what the agent remembers beyond the current percept.
• Utility — a number expressing how much a state or outcome is preferred.

The mental model

The six dimensions, and what each one costs the agent.

Observable or partially observable. If the sensors give the whole relevant state, the current percept is enough to decide and the agent needs no memory. If they do not — because sensors are noisy, limited, or cannot see through walls — the agent must maintain an internal picture of what it believes the state to be, updated by each new percept. This is the single dimension that most often forces a redesign, because partial observability is the default in the physical world and full observability is the exception people accidentally assume.

Deterministic or stochastic. If an action's effect is certain, the agent can plan a sequence of actions and execute it. If it is not, plans must be conditional, monitored and repaired, and the agent must reason about probabilities rather than certainties. There is a third case worth naming: an environment can be deterministic in principle and unpredictable to the agent because it cannot observe enough — from the agent's point of view that is stochastic, and it should be treated as such.

Episodic or sequential. In an episodic environment each decision is independent — classify this image, then the next, with no consequences carrying over. In a sequential one, the current action affects everything that follows, so the agent must think ahead. Sequential is dramatically harder, because a locally good action can be globally bad and only lookahead reveals it.

Static or dynamic. If the world waits while the agent thinks, deliberation is free and only the answer matters. If it does not, the time taken is part of the decision, and a correct answer that arrives late is a wrong answer. There is a middle case, semidynamic, where the world does not change but the agent's score does — a game against a clock.

Discrete or continuous. Distinct states, times and actions can be enumerated and searched. Continuous ones cannot, so they need different machinery — discretisation, or methods that work directly in continuous spaces.

Single-agent or multi-agent. The last one is what turns a puzzle into a game. If other agents are present, their behaviour depends on yours, and the distinction between competitive and cooperative decides everything about what strategy means. A puzzle has an optimal solution; a competitive game has an optimal strategy, which is a different kind of object — and randomising your behaviour, which is never useful against a puzzle, can be the best available strategy against an opponent.

Real tasks sit awkwardly between the clean categories, and saying where is the point of the exercise. Chess is fully observable, deterministic, sequential, semidynamic under a clock, discrete and multi-agent. Driving is partially observable, stochastic, sequential, dynamic, continuous and multi-agent — the hardest value on every dimension, which is exactly why it is hard. Medical diagnosis is partially observable, stochastic, sequential when treatment continues over time, and arguably multi-agent since the patient acts too. Sorting incoming mail is close to episodic and fully observable, which is why it is a much easier problem than it first appears.

The four agent structures each add one capability the previous lacks.

A simple reflex agent maps the current percept straight to an action by condition-action rules. It has no memory. It is adequate only when the environment is fully observable, because otherwise two different situations produce the same percept and it must respond identically to both — and in a partially observable world this reliably produces infinite loops, an agent oscillating between two states forever with no way to notice.

A model-based reflex agent adds internal state: a picture of the world updated by each percept, using knowledge of how the world evolves and how its own actions change it. That is what makes partial observability workable — it remembers what it cannot currently see.

A goal-based agent adds an explicit goal and considers whether an action leads toward it, which requires looking ahead. This is what sequential environments demand, and it is where search and planning enter, since deciding which action leads to the goal is exactly a search problem.

A utility-based agent adds a measure of how much each outcome is preferred, not merely whether it is a goal. That is needed for three situations a goal cannot express: conflicting goals that must be traded off, several routes to a goal that differ in quality, and uncertainty, where you must weigh how good an outcome is against how likely it is. Expected utility — value weighted by probability — is the formal content of "best expected outcome" from the rational-agent definition.

So the argument runs one way: environment to agent. Partial observability forces internal state, so at least model-based. Sequential forces lookahead, so at least goal-based. Uncertainty or conflicting objectives force utilities. Building less structure than the environment requires does not produce a slightly worse agent; it produces one that cannot succeed, and being able to make that argument before writing code is the entire point of the classification.

What you should now be able to explain or do

Name the six dimensions and say what each costs the agent. Explain why an environment can be deterministic in principle and stochastic in practice. Say why sequential is dramatically harder than episodic. Classify real tasks on all six at once and note where they sit awkwardly. Describe the four agent structures and the one capability each adds. Explain the failure mode of a reflex agent under partial observability. Argue from a classified environment to the minimum agent structure required.

Check yourself