Agent Memory Architecture: Designing State, Retrieval, and Forgetting for Production Agents

Why Agent Memory Breaks in Production

Most agents demo beautifully and then degrade in production. In our experience, the model rarely causes the failure—memory architecture does. Teams bolt a vector store onto a chat loop, stuff retrieved chunks into the context window, and hope the agent "remembers." For a single-user, short-session demo, that's fine. At scale—long sessions, many users, strict latency and cost budgets—it collapses.

The fix is to treat agent memory as a first-class system, not a feature. That means designing three pillars together: state (what the agent knows right now), retrieval (what it pulls back at the right moment), and forgetting (what it discards, decays, or supersedes). Get all three right and you get agents that are reliable, cost-controlled, and auditable. Skip any one and you get the familiar production symptoms: lost context, bloated prompts, and stale facts that quietly poison reasoning.

This guide breaks down each pillar, shows how they fit into a reference architecture, covers the write path and evaluation that most guides skip, and ends with a readiness checklist you can run against your own stack.

The Demo-to-Production Gap

Demos run on short sessions with curated context. A human picks the inputs, the conversation lasts ten turns, and nobody measures token cost. Production is the opposite: long-horizon sessions, thousands of concurrent users, and SLAs on latency and spend. The context window becomes a scarce resource. Facts arrive from multiple sources and contradict each other. A single session may span days. Memory that was an afterthought in the demo becomes the load-bearing wall—and it was never engineered to bear load.

The Three Failure Modes: Amnesia, Bloat, and Drift

Almost every production memory problem maps to one of three failure modes:

Amnesia — the agent loses context across sessions or after a restart. It re-asks questions the user already answered and forgets task state mid-workflow. This is a state failure.
Bloat — stale or irrelevant context floods the prompt. Reasoning quality drops, latency climbs, and token costs balloon. This is a retrieval failure.
Drift — contradictory or outdated facts persist and get retrieved as truth. The agent confidently cites a policy that changed last quarter. This is a forgetting failure.

Memory failures are rarely model failures. They are architectural failures—state that isn't persisted, retrieval that isn't ranked, and forgetting that isn't designed.

Pillar 1 — State: Designing What the Agent Knows Right Now

Working State vs. Persistent State

Separate two things that teams often conflate. Working state is ephemeral: the context window, the scratchpad, the current tool outputs. It lives for one reasoning step or one session and is cheap to discard. Persistent state is durable: user profiles, task progress, commitments, and decisions that must survive restarts and span sessions.

The design rule: working state is optimized for speed and token efficiency; persistent state is optimized for durability, schema stability, and auditability. Don't store durable facts in the context window and hope they survive. Don't persist every scratchpad token to a database.

State Schemas and Typed Memory Records

Treat every memory as a typed, versioned record, not a free-text blob. A minimal schema covers:

id — stable identifier
type — profile, task, preference, fact, decision
content — the payload, structured where possible
source — which tool, user, or document produced it
timestamp — when it was written
confidence — how much to trust it (see below)
supersedes — the id of any record this one replaces
ttl — when it should expire
schema_version — for safe migration

Typed records make retrieval precise, deletion surgical, and audits tractable. They also let you validate writes at the boundary instead of discovering malformed memory at inference time.

On confidence: do not rely on the model to self-report a trust score—those numbers are poorly calibrated. Derive confidence from objective signals: source reliability (authoritative system vs. free-text user chat), corroboration (how many independent sources assert the same fact), and recency. Then use it: gate factual answers on a confidence threshold so low-trust records inform tone and context but never get stated as fact.

Checkpointing and Session Continuity

Persist agent state at decision boundaries—after a tool call, before a handoff, at the end of a turn. Checkpoints enable resumability (the agent picks up where it left off) and replay (you can reconstruct exactly what the agent knew when it acted). Make writes idempotent so retries and duplicate events don't corrupt state. A checkpoint that can't be replayed is a checkpoint you can't debug.

Pillar 2 — Retrieval: Getting the Right Context at the Right Time

Memory Tiers: Short-Term, Episodic, Semantic, Procedural

Not all memory deserves the same storage or the same retrieval path. Design four tiers:

Short-term — the active session: recent turns, current task frame.
Episodic — past interactions and outcomes: what happened, when, and how it resolved.
Semantic — durable facts and knowledge: policies, entity attributes, domain knowledge.
Procedural — how to do things: tool-use patterns, playbooks, learned workflows.

Each tier has different access patterns, retention rules, and trust levels. Mixing them into one undifferentiated vector index is the root of most retrieval bloat.

Retrieval Strategies: Hybrid Search, Reranking, and Recency Weighting

Pure vector search is not enough. Combine keyword and semantic search (hybrid) so exact identifiers and entity names aren't lost to embedding drift. Then rerank the candidate set with a cross-encoder or a lightweight model to push the truly relevant records to the top. Reranking adds per-query latency and cost—budget for it, batch where possible, and cache reranker scores for repeated queries.

Finally, apply recency weighting—but treat it as a tie-breaker, not a truth signal. A preference stated yesterday usually outranks one from six months ago, unless the older record is flagged durable or the newer one hasn't been corroborated. Implement recency as a configurable per-type half-life (preferences decay fast; identity facts don't decay at all) rather than one global decay curve. And remember: explicit supersession always beats recency. If a record was invalidated by a later authoritative write, it should never be retrieved regardless of age—recency weighting is for ranking among valid records, not for deciding validity.

Pillar 3 — Forgetting: Deciding What to Discard, Decay, and Supersede

Forgetting is the pillar teams skip, and it's the one that causes drift. An agent that only adds memory eventually drowns in it. Forgetting has three distinct mechanisms, and conflating them is a common mistake:

Expiry — records past their ttl are dropped. Right for short-lived facts (a temporary address, a session-scoped constraint).
Decay — records lose retrieval weight over time without being deleted. Right for preferences and episodic context that should fade but remain recoverable.
Supersession — a new record explicitly invalidates an old one via the supersedes field. Right for facts that change (a policy update, a changed plan). The old record is retained for audit but excluded from retrieval.

Design forgetting at write time, not as a cleanup job. Every record should answer "when and how does this stop being true?" If you can't answer that, you don't yet understand the record.

The Write Path: Where Memory Systems Actually Fail

Most guides focus on retrieval and skip the write path—which is where production memory quietly breaks. Three things must happen when a memory is written:

Extraction — at the end of a turn or task, extract candidate memories from the interaction rather than dumping raw transcripts. Structured extraction (typed records, not free text) is what makes the rest of the pipeline possible.
Deduplication — key records on (type, entity, attribute) so repeated assertions update rather than accumulate. Without dedup you get five copies of the same preference and retrieval noise.
Consolidation — periodically merge episodic events into semantic facts and log the supersession. "User changed their shipping address three times this month" is an episodic sequence; "current shipping address = X" is the consolidated semantic fact that should actually be retrieved.

The write path is also your best defense against memory poisoning—malicious or erroneous inputs that get persisted and later retrieved as truth. Validate at the boundary, attribute every write to a source, and never let unverified free-text input overwrite a high-confidence record.

Evaluating and Observing Memory

You cannot operate what you don't measure. Memory quality needs both offline evaluation and runtime observability.

Offline: build retrieval eval sets with known query→relevant-record pairs and track recall and precision at k. Add contradiction tests (does the agent prefer the current fact over the superseded one?) and poisoning tests (does a malicious write get retrieved?). These catch drift before users do.

Runtime: track retrieval hit rate (how often retrieval returns anything useful), stale-fact rate (retrieved records later found superseded), and prompt token budget utilization (retrieved tokens vs. available budget). A rising stale-fact rate is your earliest signal that forgetting isn't working.

Reference Architecture

The pillars compose into a single flow:

Write path → extract typed records → deduplicate → validate → persist with source, confidence, ttl, supersedes. Retrieval path → route query to relevant tiers → hybrid (keyword + semantic) candidate generation → rerank → filter by validity (expiry, supersession, confidence threshold) → apply recency as tie-breaker → assemble within token budget. Forgetting path → scheduled expiry of ttl records → decay weighting → supersession on conflicting authoritative writes → consolidation of episodic into semantic.

[ILLUSTRATION: Four-tier memory stack—short-term, episodic, semantic, procedural—feeding a hybrid retrieval pipeline with reranking and validity filtering, alongside a separate write path handling extraction, dedup, and consolidation.]

Production Readiness Checklist

Run this against your stack before you ship:

If you can check every box, your memory is engineered. If you can't, you've found the next production incident before it happens.