Agent Memory: Write, Read, Update, Forget

From Function Calling to the Agent Loop’s agent remembers only its transcript, and only until the transcript is cut. An assistant that should know next week what a user told it today needs more: somewhere to put what it learns, and a rule for what to bring back. This post treats memory as what it is in every working system, storage plus a retrieval policy, and measures how much the policy matters.

Table of contents

What “memory” means for an agent

The context window is working memory: everything the model can use on this call, and nothing else. Everything outside it is storage, and the model only benefits from what a retrieval step puts back into the window. Borrowing from cognitive science (Sumers et al., 2024), stored memory comes in kinds:

• Episodic: what happened, a log of events and conversations (“on Monday the user asked about RoPE”).
• Semantic: facts distilled from episodes (“the user is learning German”).
• Procedural: how to do things, held in the model’s weights and the agent’s code, and in instructions loaded on demand (Giving Agents Tools and Knowledge’s Skills).

The four operations

Each needs a rule, and each rule is a design decision:

• Write: what to store. Everything is simple and expensive, and makes retrieval harder. Most systems store summaries or extracted facts, which needs a model call and can lose detail.
• Read: what to bring back, given a limited budget of tokens. This is retrieval (Part IX: Retrieval, from Vector Search) with one addition: when something was stored matters too, whenever facts change.
• Update: what to do when a new fact contradicts an old one. Appending both and hoping the reader sorts them out is the default, and the experiment below measures what it costs.
• Forget: what to delete, for size, for relevance, or because a user asked. A memory that can’t forget can’t comply with a deletion request.

Generative Agents: recency, importance, relevance

Park et al. (2023) gave each of 25 simulated characters a memory stream, a list of observations, and retrieved by a score with three parts, each min-max scaled to $[0, 1]$ over the stream and summed with equal weights:

• Recency: decays exponentially, in the paper with time since the memory was last used (0.995 per game hour); below, with time since it was written (0.99 per turn).
• Importance: a 1-to-10 rating the model gives each memory when it is written (“brushing teeth” low, “a breakup” high).
• Relevance: cosine similarity between the memory’s embedding and the current query’s (Embeddings and Contrastive Learning).

Relevance alone finds what is about the question; recency breaks ties toward what is current; importance keeps significant memories from being crowded out by trivia. The paper’s write policy is reflection: when the importance of recent memories sums past a threshold, the model asks itself questions about them and stores its answers, higher-level inferences pointing back to their evidence, as new memories that retrieval can return.

MemGPT: memory as an operating system

MemGPT (Packer et al., 2023) borrows from virtual memory instead. The context is main memory: system instructions, a small editable block of facts (its working context, which the model rewrites with a replace call), and a queue of recent messages. Outside it are recall storage, every past message, searchable, and archival storage, any text, searched by embedding, like disk. The model moves information with function calls, and the system warns it when the context passes about 70% full; when it fills, the oldest messages are evicted and replaced by a recursive summary. Memory management becomes something the agent does, not something done to it. Context Engineering’s memory tool, files the agent reads and writes, is the same idea in its simplest form.

An experiment: when facts change

Retrieval policies are easiest to compare without a language model in the loop: measure whether the policy puts the right memory in front of the model, which is the most any answer could use. Synthetic conversations of 50 turns, each with 8 facts about the user mixed into small talk, 3 of them changed later in the conversation; at the end, one question per fact. 250 conversations, over five random seeds. Importance here isn’t rated by a model: facts get 8 and small talk 3, a perfect rater, so two extra rows each drop one of the three terms. retrieval.py builds the conversations, embeds every turn and question with a small sentence encoder (all-MiniLM-L6-v2), and records what each policy shows. The policies themselves:

def shown(policy, turns, importance, E, q, k):
    now = len(turns)
    rel = E @ q
    if policy == "last k turns":
        return list(range(now - k, now))
    if policy == "most relevant k":
        return list(np.argsort(-rel)[:k])
    recency = 0.99 ** (now - 1 - np.arange(now))                     # Generative Agents' three scores
    norm = lambda x: (x - x.min()) / (x.max() - x.min() + 1e-9)
    use = {"recency + importance + relevance": (1, 1), "recency + relevance": (1, 0),
           "importance + relevance": (0, 1)}[policy]
    score = use[0] * norm(recency) + use[1] * norm(importance) + norm(rel)
    return list(np.argsort(-score)[:k])

Full script: retrieval.py

                                    k  current only   both  outdated only  neither
last k turns                        1            2%     0%             0%      98%
last k turns                        3            5%     0%             0%      95%
last k turns                        5            9%     0%             0%      91%
most relevant k                     1           71%     0%            29%       0%
most relevant k                     3           62%    35%             2%       0%
most relevant k                     5           62%    36%             1%       0%
recency + importance + relevance    1           87%     0%             4%       9%
recency + importance + relevance    3           63%    36%             0%       0%
recency + importance + relevance    5           62%    38%             0%       0%
recency + relevance                 1           60%     0%             3%      37%
importance + relevance              1           71%     0%            29%       0%

worked: "What is the user's job?"
  turn 25 'Actually, things changed: I work as a data engineer.'   recency 0.45 + importance 1.00 + relevance 0.84 = 2.29
  turn 41 'Actually, things changed: My pet is a hamster.'         recency 0.80 + importance 1.00 + relevance 0.30 = 2.10
  turn  4 'I work as a teacher.'                                   recency 0.06 + importance 1.00 + relevance 1.00 = 2.06

Three of the 8 facts change, so 37.5% of questions concern a fact with an old and a new version.

• A recent window misses almost everything. The facts were stated turns ago; the last 5 turns hold the answer 9% of the time. This is memory as a chatbot with a short context has it.
• Relevance finds the fact, but not which version. With room for one memory, relevance alone shows only the outdated version for 29% of all questions, more than three in four of the changed facts. The original statement is usually a little more similar to the question than the update, partly because the update’s “Actually, things changed:” adds words the question doesn’t share.
• Recency picks the version; importance keeps chatter out. The three-part score shows only the outdated version for 4%. The worked example is the mechanism: relevance prefers the original “I work as a teacher” (1.00 against 0.84), and recency (0.45 against 0.06) puts the update first. The ablations split the work: without importance, recency pulls recent small talk to the top and 37% of questions get neither version; without recency, importance is the same for every fact and changes nothing.
• It has a cost. With the three-part score, 9% of questions get neither version, against none with relevance alone: a recent update about something else can outrank an older fact, as the worked example’s second row does, a pet ahead of a job. The total error still falls, from 29% to 13%, but it is a trade, and with a real, noisier importance rater it would be a worse one.
• A bigger budget turns errors into conflicts. With room for 3 or 5 memories, both the relevance-only and the three-part policy show both versions for 35 to 38% of questions, nearly every changed fact, and the three-part score is no better than relevance alone: it changes which memory comes first, not which are shown. The model must then work out which is newer, which it can do only if memories carry their dates or their order.

The update rule is therefore the one to design deliberately. The cheapest fix is at read time: show retrieved memories in the order they were written, with timestamps. The thorough fix is at write time. Production systems do this: Mem0 has a model decide, for each new fact, whether to add it, update or delete an existing one, or do nothing; Zep stores facts in a knowledge graph whose edges carry validity intervals, so a superseded fact is marked invalid rather than lost. This experiment shows the problem, not those fixes. The benchmarks for the real version of it are LongMemEval, whose knowledge-update questions are exactly this case, and LoCoMo, over very long conversations.

Forgetting, and privacy

Memory stores personal information by design, so the usual rules apply, and some apply harder. Store only what the purpose needs. Tell users what is remembered, and let them see, correct and delete it; deletion must reach every copy, including summaries and embeddings derived from the original. Separate memories by user, so retrieval can never return one user’s facts to another, which a shared vector index will do unless every query filters by owner (Vector Search). And treat stored memories as untrusted input when they are read back: a memory written from a web page can carry instructions (see Guardrails and Prompt Injection), and unlike an injected web page it comes back in every later session that retrieves it (Chen et al., 2024, showed a few poisoned memories are enough).

Where this leads

• Beyond Vector RAG lets an agent decide what to retrieve, step by step.
• Giving Agents Tools and Knowledge loads procedures and tools on demand: Skills and MCP.

Takeaways

• Context is working memory; everything else is storage plus a retrieval policy, and the policy decides what the model can use.
• Each operation needs a rule: what to write, what to read within a budget, how to update contradictions, and what to forget.
• On conversations whose facts change, a recent window found 9% of facts; relevance found nearly all of them but, with a budget of one memory, showed only the outdated version for 29% of questions. Generative Agents’ score cut that to 4%, at a cost: 9% of questions then got neither version, and it needed a perfect importance rater to keep small talk out.
• With larger budgets, retrieval shows both versions; fix it at read time with timestamps and order, or at write time by replacing or marking what a new fact contradicts.
• Memory is personal data: scope it per user, make it visible and deletable, and treat what it returns as untrusted input.

References

Start with Generative Agents for the memory stream and its score, then MemGPT for memory managed by the agent itself; LongMemEval is where to measure a memory system on changing facts.

• Park, O’Brien, Cai, Morris, Liang and Bernstein, Generative Agents: Interactive Simulacra of Human Behavior (UIST, 2023): the memory stream, its recency, importance and relevance score, and reflection.
• Packer, Wooders, Lin, Fang, Patil, Stoica and Gonzalez, MemGPT: Towards LLMs as Operating Systems (2023): memory tiers managed by the agent through function calls.
• Chhikara et al., Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (2025): write-time add, update, delete.
• Rasmussen et al., Zep: A Temporal Knowledge Graph Architecture for Agent Memory (2025): facts in a knowledge graph with validity intervals.
• Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR, 2025); Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents (ACL, 2024), LoCoMo: the two benchmarks for memory over long conversations; LongMemEval has the knowledge-update questions.
• Chen, Xiang, Xiao, Song and Li, AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS, 2024): a few poisoned memories are enough to steer an agent.
• Sumers, Yao, Narasimhan and Griffiths, Cognitive Architectures for Language Agents (TMLR, 2024): the memory taxonomy.