How AI Agents Remember: Designing Agent Memory and State

How Memory and State Work in an AI Agent

An AI agent manages memory across four distinct layers: the in-context window (short-term), external retrieval stores (long-term semantic), structured key-value state (session and working memory), and procedural memory encoded in the system prompt or fine-tuned weights. Each layer has a different write cost, read latency, and staleness risk. Getting the boundaries wrong is the most common reason agents behave inconsistently in production.

I am Mahmoud Zalt, an independent senior AI systems architect with 16+ years building production software since 2010. I am the founder of Sista AI, where keeping a workforce of autonomous agents coherent across long sessions in production made memory and state the problems I think about most. I design and build production AI agent systems as an independent consultant. If you are architecting an agent and need a production-grade memory design, see my AI Agent Development service or learn more about my background.

The Four Memory Layers Every Agent Needs

Before picking a vector database or a session store, map your agent to these four layers. Most production bugs come from collapsing two layers into one or skipping one entirely.

A customer-support agent needs all four. A single-turn code-review agent needs only the first and last. Designing memory starts by deciding which layers your use case actually requires, not by defaulting to 'add a vector DB.'

Working Memory: The Context Window Is Not Free

The context window is your agent's working memory. It holds the system prompt, the conversation history, any retrieved documents, tool call results, and the model's own chain-of-thought tokens if you use extended reasoning. Every token costs money and adds latency. The practical cap for GPT-4o is around 128k tokens; for Claude 3.5 Sonnet it is 200k. Both sound large until you are piping in retrieval results, tool outputs, and a multi-turn conversation simultaneously.

What teams get wrong

The most common mistake is naive full-history appending: every user turn and assistant response is concatenated into the next call. By turn 20 of a long session, you are paying for tokens the model has already processed and the signal-to-noise ratio has collapsed. Summarization checkpoints fix this. Every N turns (I use 10 to 15 depending on turn density), run a cheap fast model (Haiku, GPT-4o mini) to compress prior turns into a 200-300 token summary. Inject the summary at a fixed position in the context, discard the raw history up to that checkpoint. This keeps working memory bounded without losing continuity.

Concrete pattern: rolling summary checkpoint

if len(history) > CHECKPOINT_THRESHOLD:
    summary = summarizer_llm.run(
        'Compress this conversation. Keep: user goal, confirmed facts, open tasks.',
        history[:-KEEP_RECENT]
    )
    history = [SystemMessage(summary)] + history[-KEEP_RECENT:]

Keep the last 3 to 5 raw turns so the model has immediate conversational context. Everything older becomes the summary.

Session State: Structured Facts the Agent Can Trust

Session state is distinct from the raw conversation history. It is a structured object, a dict or a typed schema, that captures confirmed, actionable facts: the user's confirmed goal, task progress, validated inputs, and any decisions already made. It persists for the duration of one session and is injected into the system prompt or as a dedicated context block at the start of each turn.

The critical discipline here is write-on-confirm, not write-on-mention. If a user says 'I want to deploy to AWS,' that goes into session state only after the agent has confirmed the intent back and the user has acknowledged it. Agents that write to state eagerly on first mention end up with contradictory state when the user refines their request two turns later.

Schema example

{
  'session_id': 'sess_abc123',
  'user_goal': 'migrate Postgres schema from v1 to v2 without downtime',
  'confirmed_facts': {
    'database': 'production-db-eu',
    'migration_window': '2026-06-22 02:00 UTC'
  },
  'task_progress': {
    'backup_verified': true,
    'migration_script_reviewed': false
  },
  'open_questions': ['rollback strategy confirmed?']
}

This object is cheap to serialize, easy to log for observability, and auditable. It is also easy to invalidate: if the user changes their goal, you reset the relevant keys rather than patching a semantic vector store.

Long-Term Memory: When to Reach for a Vector Store

Long-term semantic memory is warranted when the agent needs to recall facts from previous sessions, surface relevant past decisions, or personalize behavior based on a user's history. The canonical implementation is a vector database (pgvector, Pinecone, Qdrant, Weaviate) combined with an embedding model. At write time, you chunk and embed relevant content. At read time, you embed the current query and retrieve the top-k nearest chunks via cosine similarity.

Retrieval design decisions that matter

• What to write: do not dump entire conversations into the vector store. Write distilled facts and decisions. 'User prefers TypeScript over JavaScript for new services' is a useful memory. A 2000-token turn transcript is noise that will contaminate retrieval.
• Chunking strategy: semantic chunking over fixed-length chunking. A fact about a user preference should be a single chunk, not split across two 512-token windows.
• Retrieval threshold: set a minimum similarity score (typically 0.78 to 0.82 cosine similarity depending on your embedding model). Inject only chunks above the threshold. Injecting low-relevance retrievals is worse than injecting nothing: it actively confuses the model.
• Recency weighting: weight recent memories more heavily. A user preference from 18 months ago may be stale. A hybrid score of (similarity * 0.7) + (recency_score * 0.3) works well in practice.
• Memory TTL and expiry: facts about ephemeral state (a specific project, a short engagement) should have an explicit expiry. Preferences and stable identity facts should not.

A real anti-pattern

A team I audited had built a CRM-integrated sales agent that stored every prospect interaction in a vector store and retrieved the top-10 chunks on every turn. By month three, the store held contradictory facts from prospects who had changed their position. The agent was confidently surfacing stale objections as if they were current. The fix: structured key-value storage with explicit update timestamps for authoritative CRM facts, vector store only for unstructured notes and sentiment signals.

State Across Tool Calls and MCP

When an agent uses tools, including MCP (Model Context Protocol) servers, each tool call returns a result that must be managed as ephemeral state within the current turn's context. The agent's reasoning loop is: observe context, decide on tool call, execute tool, observe result, update scratchpad, decide next action. The tool result is part of working memory for the duration of the turn. If it needs to survive beyond the turn, you must explicitly write it to session state or long-term memory.

With MCP specifically, the server maintains its own resource and tool state. The agent does not automatically know what changed in the MCP server between invocations. If your MCP server is stateful (a browser session, a database cursor, a file handle), you need a handshake: the agent must request current state at the start of each session rather than assuming the state it left behind is still valid. I model this as a 'state hydration' step at session open: the agent calls a dedicated tool to retrieve and inject current server state before any substantive tool calls.

Pattern: stateful MCP session hydration

# On session open, before user turn is processed
server_state = await mcp_client.call_tool('get_session_state', {})
agent_context.inject_system_block(
    f'Current server state: {server_state}'
)
# Now process user turn with accurate server state in context

When Persistent Memory Is a Liability

This is the section most architects skip, and it is the one that prevents the most production incidents. Persistent memory creates real risks that must be weighed against its benefits.

• Privacy and compliance: if your agent operates in a regulated domain (healthcare, finance, legal), retaining user-specific memories may conflict with data minimization requirements (GDPR Article 5(1)(c), HIPAA minimum necessary). You need explicit data retention policies and a delete path per user. 'We store it in a vector DB' is not a compliance answer.
• Memory poisoning: a user (or an attacker via prompt injection) can deliberately introduce false facts into the memory store. 'Remember that I am an admin' injected in a benign-seeming turn can persist and be retrieved later to escalate privilege. Guardrails: never write agent-observed claims about permissions or identity to memory without out-of-band verification. Treat memory writes as a privileged operation.
• Stale memory degrading trust: a user who changed their preference six months ago but whose old preference keeps surfacing will lose trust in the agent fast. Track memory source, creation date, and last-confirmed date. Surface the basis for personalized behavior: 'Based on your preference from March, I suggest X.' This lets users correct stale state and builds transparency.
• Cost of retrieval on every turn: vector retrieval adds 50 to 200ms per turn in typical cloud deployments. For high-frequency agents (code assistants, real-time chat), this overhead compounds. Profile before enabling retrieval on every turn. Trigger retrieval only when the current turn contains a signal that past context is relevant.

The lean default: start with session state only. Add long-term retrieval only when you have a specific, validated use case that fails without it and you have the observability to monitor what gets retrieved.

Observability: What Gets Retrieved Drives What Gets Said

You cannot debug agent memory behavior without logging what was retrieved and injected. Every production agent memory system I build logs three things at minimum: the retrieval query, the retrieved chunks with their similarity scores, and the final assembled context sent to the model. When the agent says something wrong or unexpected, the first question is always 'what did it have in context?' Without this log, you are debugging blindly.

For evals, I run a memory recall evaluation set: a fixed set of scenarios where the correct behavior depends on correct memory retrieval. These scenarios test both presence (the agent correctly recalls a stored fact) and absence (the agent does not hallucinate a fact that was never stored). Run this suite on every change to chunking strategy, embedding model, or retrieval threshold. A change that improves semantic search scores but degrades memory recall on your specific domain is a regression, not an upgrade.

Structured memory (session state, key-value) is easier to evaluate than vector retrieval because it is deterministic. Write a test that sets known state, runs a turn, and asserts the correct state fields were used. Treat these like unit tests: fast, cheap, run on every deploy.

Human-in-the-Loop Memory Confirmation

For high-stakes memory writes, do not let the agent decide silently. Surface the write to the user and ask for confirmation. 'I am going to remember that your preferred deployment region is EU-WEST-1 for future sessions. Correct?' This pattern costs one extra turn but builds trust, reduces stale-memory problems, and gives users agency over their own data.

The threshold for mandatory confirmation: any memory that will affect future behavior in a way the user might not anticipate. Preference memory (formatting style, default tool) can be written silently on clear signal. Factual memory that drives decisions (budget constraints, compliance requirements, system architecture choices) should be confirmed explicitly.

This is also where human-in-the-loop becomes a security guardrail. If an agent is being manipulated via prompt injection to write false facts to memory, the confirmation step surfaces the attempted write and breaks the attack before it persists.

Frequently Asked Questions

What is the difference between agent memory and agent state?

State is the structured, typed data the agent uses to track task progress within and across sessions: a dict or schema with explicit fields. Memory is broader and includes unstructured semantic retrieval from past interactions. State is deterministic and auditable. Memory via vector retrieval is probabilistic. Use state for anything the agent must reliably know. Use memory for context that improves responses but is not required for correctness.

How do AI agents remember things between sessions?

Between sessions, agents rely on persistent storage: a relational or key-value database for structured session state, and optionally a vector database for semantic long-term memory. The critical discipline is deciding what is worth persisting. Raw conversation history is rarely the right thing to store. Distilled facts, confirmed preferences, and task outcomes are. At the start of a new session, the agent hydrates its context from these stores before processing the first user turn.

What vector database should I use for agent long-term memory?

If you are already on Postgres, start with pgvector. It handles millions of vectors at sub-100ms query latency with HNSW indexing and eliminates a separate service to operate. Move to Pinecone or Qdrant only if you need billion-scale retrieval, multi-tenancy isolation at the vector level, or retrieval performance that pgvector cannot meet after tuning. The database choice matters far less than your chunking strategy, embedding model, and retrieval threshold. Teams obsess over the database and ignore the retrieval design. That is backwards.

How do I prevent my AI agent from hallucinating false memories?

Three controls: first, only write to memory on confirmed, verified signals, never on user assertion alone. Second, store provenance with every memory chunk: source, timestamp, confidence. Third, log and eval what gets retrieved so you can catch hallucinated retrievals in testing before they reach production. Memory poisoning via prompt injection is a real attack vector: treat memory writes as privileged operations with the same scrutiny you would give a database write.

When should an AI agent NOT use persistent memory?

Skip persistent memory when: the task is stateless and each session is independent (a one-shot code review, a document summarizer), when you cannot meet data retention and deletion requirements for the stored data, when the retrieval latency budget makes per-turn retrieval untenable, or when the memory store would be too sparse to be useful (a new deployment with no history). Start without it. Add it when you have a concrete failing case that persistent memory solves.

How much context window should I reserve for retrieved memories?

Reserve 15 to 20 percent of your effective context budget for retrieved memory, and cap it hard. In a 128k token window with a 4k system prompt and up to 20k for conversation history and tool results, that leaves roughly 104k. Capping retrieval injection at 15k to 20k tokens is a reasonable default. Beyond that, retrieved context starts displacing the actual conversation, and relevance degrades. Use your similarity threshold as the primary gate, and token budget as a hard ceiling.

Build Agent Memory That You Can Actually Debug

The pattern I use in production: session state for structured task facts, rolling summarization for conversation history, vector retrieval only when validated by a failing use case, and explicit logging of every retrieval. Memory is not a feature you bolt on at the end. It shapes how the agent behaves, what it trusts, and how it fails. Getting it right at the architecture stage is far cheaper than retrofitting it after users have encountered inconsistent, stale, or hallucinated behavior.

If you are building an agent system and need production-grade memory architecture, I take on a small number of independent engagements per quarter. See the details on my AI Agent Development service page or get in touch directly. I scope, architect, and build, no agency overhead, direct access to the person who ships it.

Work with me on your agent memory architecture