Agent Memory Architecture: What to Store, What to Forget, and What It Costs
Learn what to store, forget, and budget in agent memory architecture. Build short-term, long-term, episodic, and semantic tiers that scale in production.
Why Memory Is the Hardest Part of Production AI Agents
Every team building agents hits the same wall. An agent that remembers everything becomes expensive, slow, and eventually incoherent—retrieving stale facts that contradict what it learned yesterday. An agent that forgets everything is cheaper and faster but also useless: it re-asks the same questions and never accumulates expertise.
The instinct is to treat this as a storage problem. Add a vector database, embed the conversation history, retrieve the top-k chunks, done. That works in a demo and fails in production, because agent memory is a cost and quality tradeoff, not a storage problem. Every memory you keep costs tokens on retrieval, dollars in infrastructure, and milliseconds of latency—and it can actively degrade answer quality if it crowds out better context.
This article breaks the problem into the three decisions that actually determine whether an agent's memory helps or hurts: what to store, what to forget, and what it costs.
What Agent Memory Actually Is (and Isn't)
Chat history is not memory. Neither is a vector store full of embedded transcripts. Both are raw material. Memory is the deliberate architecture layer that decides what enters the agent's context, in what form, and when it leaves. It has four jobs: capture, retrieval, ranking, and eviction. Teams that build all four get reliable agents. Teams that build only capture and retrieval get agents that slowly poison themselves.
The most common misconception is that "just add a vector database" constitutes a memory strategy. It doesn't, for three reasons:
- Similarity is not relevance. Similarity search retrieves what is semantically close, not what is useful right now. A past conversation about a canceled order may be highly similar to a new order question and completely irrelevant.
- Raw transcripts are noisy. Embedding full conversations and taking the top-k often returns filler, because most turns contain no durable signal. Chunking and extraction quality determine retrieval quality more than the index does.
- Nothing tracks supersession. A vector store has no notion of a fact being replaced, so outdated information competes on equal footing with current truth.
A real memory layer answers specific questions: What do we extract from an interaction? How is it summarized? What metadata governs its retrieval? When does it expire? Without those answers, you have a search index, not memory.
A vector database is to agent memory what a filing cabinet is to a brain. It holds things. It doesn't decide what matters.
A Working Taxonomy: Tiers and Content Types
The cleanest way to reason about memory is on two independent axes. Tiers describe how long information lives and where it is stored. Content types describe what kind of information it is. Production agents need both axes, because the storage and cost behavior of a tier is different from the storage and cost behavior of a content type.
| Short-term (working context) | Long-term (persistent store) | |
|---|---|---|
| Episodic (events) | Current task's recent turns and tool outputs | Summarized past interactions, decisions, outcomes |
| Semantic (facts/rules) | Task state object, active constraints | Entity facts, preferences, policies, distilled rules |
Short-Term Memory: The Working Context
Short-term memory is the active context window—everything the agent needs to complete the current task. It is the most expensive memory per token, because it is re-sent with every model call. The engineering discipline here is context management: summarizing completed sub-tasks, dropping resolved tool outputs, and keeping only the state the agent still needs. A 40-turn conversation should not mean a 40-turn prompt.
Rolling summarization—compressing older turns into a short state summary while keeping recent turns verbatim—is a common pattern, but it has a known failure mode: summarization is lossy, and chained summaries drift. Two mitigations matter. First, regenerate summaries from a pinned source of truth (the full transcript or a structured state object), not from the previous summary. Second, keep a structured state object—explicit fields for task, entities, and open questions—alongside any free-text summary, so the agent isn't reasoning from prose alone. Treat the context window as a budget to be spent, not a bucket to be filled.
Long-Term Memory: The Persistent Layer
Long-term memory persists facts, preferences, and outcomes across sessions. It typically combines a vector store for semantic retrieval with a structured database for exact lookups (user ID, account tier, stated preferences). The structured layer matters more than teams expect: many "memory" lookups are really key-value reads, and a SQL query is cheaper and more precise than a similarity search. Use vectors for fuzzy recall, structured storage for facts you can name.
Episodic vs. Semantic Content
These are the two content types that live in long-term memory, and conflating them is a common failure.
- Episodic memory records specific events: "On March 3, the user escalated a billing dispute and the agent issued a credit." It preserves context and rationale—useful for continuity, personalization, and audit.
- Semantic memory holds distilled facts and rules: "This user prefers email over phone," "Refunds above $500 require manager approval." It is compact, durable, and cheap to retrieve.
Episodic memory is richer but noisier and grows fast. Semantic memory is leaner but requires extraction work. The practical pattern is to store episodes, then distill them into semantics—and let the raw episode age out once the fact has been extracted.
[ILLUSTRATION: Two-axis memory diagram—tiers on one axis (short-term context window, long-term vector + structured stores), content types on the other (episodic, semantic)—with an episodic-to-semantic distillation flow and an eviction gate at the long-term boundary]
What to Store
Store less than you think, but store it deliberately. The highest-value memory items fall into five categories:
- Durable entity facts and preferences (semantic). Names, roles, accounts, stated preferences, constraints. These change rarely and pay off on nearly every interaction.
- Task outcomes and decisions with rationale (episodic, summarized). What was decided, why, and what the result was. One distilled paragraph beats a full transcript.
- Source references and provenance. Which document, record, or tool output a fact came from. This is what makes memory auditable and lets you invalidate it when the source changes.
- Corrections and explicit confirmations. When a user says "no, that's wrong" or "yes, that's right," that is a high-signal event worth persisting. Corrections are among the cheapest quality improvements available.
- Recurrence signals. If the same request, entity, or pattern appears repeatedly, it deserves a memory entry—recurrence is evidence of relevance that a single interaction doesn't provide.
Everything else is a candidate for not storing. Raw tool outputs, intermediate reasoning, and full transcripts are usually inputs to extraction, not memory themselves.
What to Forget
Eviction is the half of memory architecture teams skip, and it is where most "poisoning" originates. Four mechanisms cover the practical cases:
- Time-to-live (TTL). Every memory entry gets a default expiry appropriate to its type: hours for volatile task state, days for episodic summaries, longer for durable preferences. Expiry should be the default, not an exception.
- Supersession. A new fact about the same entity and attribute invalidates the old one on write. "User's plan is now Enterprise" must retire "User's plan is Pro"—not sit beside it. This requires a schema that keys facts by (entity, attribute), which is exactly what the structured layer is for.
- Confidence decay. Inferred facts (guessed from behavior) should carry lower weight and decay faster than facts the user stated explicitly. Not all memories are equally trustworthy.
- Hard caps. Bound the total number of memory entries per user or entity. When the cap is hit, evict lowest-value entries by a scoring function (recency × salience × confidence). Unbounded memory is unbounded cost and unbounded retrieval noise.
Contradiction handling deserves special mention: on every write, check whether the incoming fact conflicts with an existing one. If it does, resolve it explicitly—newer wins, or escalate to the user—rather than storing both and letting retrieval arbitrate.
What It Costs
Memory is not free, and the costs are not evenly distributed. Three cost centers matter, and they trade against each other.
1. Context tokens (short-term memory). Every token in the working context is re-billed on every model call. A 10-turn conversation that grows to 2,000 context tokens and makes 10 calls pays for roughly 20,000 input tokens' worth of context across the session—before any output. This is why context management is a cost lever, not just a quality one. Prompt caching helps materially here: if the stable prefix of the context (system prompt, pinned state) is cached, the marginal cost of re-sending it drops sharply. Design the prompt so the cacheable prefix is large and stable, and keep volatile content at the end.
2. Retrieval and storage (long-term memory). Vector search and structured reads add latency and infrastructure cost per call. The tradeoff is precision vs. price: a structured key-value read is typically cheaper and more precise than a similarity search, so route named-entity lookups to SQL and reserve vector search for fuzzy recall. Every retrieved memory also consumes context tokens, so retrieval count and chunk size are cost decisions.
3. Quality cost of bad memory. This is the one teams can't see on an invoice. Retrieved stale or irrelevant memories crowd out good context and degrade answers—a cost paid in task failures and re-work, not tokens. It is the strongest argument for eviction and supersession.
A simple way to keep these honest: instrument cost per resolved task and task success rate side by side. A memory change that improves success at flat cost is a win; one that improves success but doubles cost per task needs justification; one that does neither should be reverted. Memory changes are hypotheses, and they should be measured like any other production change.
Putting It Together
The teams that get memory right build all four jobs—capture, retrieval, ranking, and eviction—and they measure the result in cost per resolved task, not in memories stored. They store semantic facts in structured form, summarize episodes before they accumulate, attach provenance to everything, and expire by default. The vector database is a component, not a strategy. The strategy is the set of decisions about what enters context, in what form, and when it leaves—and those decisions, made deliberately, are what separate a reliable agent from one that slowly forgets how to be useful.