9.4 Memory

What this is and why it exists

Agent memory is two different problems wearing one word. There is the working context right now, which fills up during a long task and has to be managed. And there is knowledge that should survive across sessions, which is retrieval applied to the agent's own history. Both are made worse by the same instinct — append everything — because an accumulating store degrades retrieval while keeping wrong and stale facts alive. What to forget is as important as what to keep.

The vocabulary

• Working context — what is in the request right now.
• Compaction — reducing the transcript to fit while preserving what matters.
• Rolling summary — a compact account of everything older than the recent turns.
• Long-term store — facts kept outside the context and retrieved when relevant.
• Episodic memory — what happened, with a time and a place.
• Semantic memory — what is true, independent of when it was learned.
• Provenance — where a remembered fact came from and when.
• Memory poisoning — wrong information written to memory and retrieved repeatedly.

The mental model

Short-term first, because every long task hits it. The transcript grows with every step — the model's reasoning, the tool calls, and above all the tool results, which are frequently the largest thing in it. Left alone it fills the window, and the cost per step rises the whole way there, since each request carries the accumulated whole.

Compaction is what you do about it, and doing it badly loses the thread. The strategy that works keeps three things and compresses the rest. Keep the task and the constraints verbatim, always, at the top — an agent that has compacted away what it was asked to do is worse than useless, and this is the commonest compaction failure. Keep the recent turns in full, because that is where the current sub-task lives. And replace everything older with a rolling summary covering what was tried, what was learned, what failed and why, and what remains outstanding.

Three practical rules. Compress tool results hardest, since a large search result or file listing was needed for one decision and never again — keep the conclusion drawn from it, not the payload. Compact on a threshold rather than at the last moment, at perhaps two thirds of the window, because compacting under pressure produces a worse summary. And keep the full transcript outside the context, written to storage, so the compaction is a view for the model and not a loss of the record — which the observability topic then depends on.

Long-term memory is retrieval applied to the agent's own history, and it is what makes memory survive a restart. Write facts to a store, retrieve the relevant ones at the start of a task or when the topic warrants, and the agent knows things a fresh session would not. Everything from the retrieval topic applies: chunking, embedding with one model, metadata filtering, access control in the filter.

What makes agent memory different is that the agent writes it, which means everything it writes can be wrong, and errors are durable.

Episodic and semantic memory are different kinds and need to stay separate. Episodic is what happened: on this date, this user asked for this, and the agent did that, and it failed. Semantic is what is true: this user prefers metric units; this account is on the enterprise plan; this endpoint requires a version header.

Mixing them produces the characteristic failure — an agent that recalls events as facts. It remembers that a deployment failed once and treats deployments as failing. It remembers a user asking about refunds and treats them as a refund case forever. Keep them in separate stores with different rules: episodes are timestamped, immutable and retrieved by similarity to the current situation; facts are current, updatable, and retrieved by subject. And derive facts from episodes deliberately rather than automatically — several similar episodes may justify a general fact, and one episode never does.

Then poisoning and staleness, which are the reason memory needs curation rather than accumulation.

Poisoning: a wrong fact written once is retrieved repeatedly, shapes later behaviour, and can be written again in a stronger form because the agent now "knows" it. It gets worse rather than decaying. And it is a security surface as well as a quality one — an injected instruction that persuades an agent to remember something false has planted it for every future session, which is a far more durable attack than one that only affects the current run.

Staleness: a fact true when stored may not be now. Prices, statuses, team members, endpoints, preferences — all of them change, and a memory store has no way to notice.

Four mechanisms handle both, and they are all ordinary engineering. Provenance on every entry: where it came from, when, and from which run — so a bad source can be traced and everything it produced removed at once. Expiry by kind: a stated preference may last a year, an account status a day, an in-progress state an hour; write the expiry when you write the fact rather than hoping to remember later. Verification for anything consequential: check the current value from the system of record rather than trusting the remembered one, and keep memory as a hint about where to look rather than as an authority. And a write policy — not everything an agent notices deserves to be remembered. Write when a fact is stable, general and would change future behaviour; do not write speculation, single observations, or anything derived from content the agent did not verify.

The rule to take away. Memory that only grows is a liability that only grows with it. Design the forgetting at the same time as the remembering — expiry, correction, provenance and a way to delete an entire source's contribution — because retrofitting deletion onto a store that has been accumulating for six months is a project, and being asked to remove a user's data from it is a Tuesday.

What you should now be able to explain or do

Distinguish the two memory problems. Compact a transcript keeping the task, the recent turns and a rolling summary, compressing tool results hardest and acting on a threshold. Keep the full transcript outside the context. Apply the retrieval rules to a long-term store. Separate episodic from semantic memory and say what mixing them produces. Derive facts from repeated episodes rather than single ones. Explain poisoning as both a quality and a security problem. Implement provenance, expiry by kind, verification and a write policy. Design forgetting alongside remembering.

Check yourself