Agent Memory Architectures: Short-Term, Long-Term, and Temporal Memory Without Drowning Your Context Window

Your agent solved the user's problem on Monday. On Thursday the user returns and the agent asks for the same account ID, the same preferences, the same constraints. Or worse: it confidently acts on a preference the user changed two weeks ago.

Both failures come from the same place: "memory" was bolted on as a single vector store, with no policy about what to write, what to retrieve, or what to forget.

Memory is three problems, not one

Layer	What it holds	Lifetime	Where it lives
Working memory	Current conversation, tool results, scratchpad	One run	The context window
Episodic memory	What happened: past sessions, outcomes, failures	Weeks to months	Event log + summaries
Semantic memory	Distilled facts: preferences, entities, rules	Until contradicted	Key-value / vector / graph

Treat these separately. Working memory is a context-engineering problem (compaction, trimming). Episodic and semantic memory are storage-and-retrieval problems with a write policy and a read policy. Most production bugs are in the write policy.

The write path: extract, don't dump

Storing raw transcripts and hoping retrieval sorts it out gives you noisy recall and a growing bill. Instead, run an extraction step after each session (or every N turns) that proposes memory operations against what you already know:

• ADD a new fact
• UPDATE a fact that changed
• DELETE a fact that is now false
• NOOP when nothing durable was said

This ADD/UPDATE/DELETE/NOOP pattern is the core idea in the Mem0 paper (arXiv 2504.19413). The important part is that the extractor sees existing memories, so it can reconcile instead of piling up contradictions.

from pydantic import BaseModel
from typing import Literal
from openai import OpenAI

client = OpenAI()

class MemoryOp(BaseModel):
    op: Literal["ADD", "UPDATE", "DELETE", "NOOP"]
    memory_id: str | None = None   # required for UPDATE / DELETE
    text: str | None = None        # required for ADD / UPDATE
    reason: str

class MemoryPlan(BaseModel):
    ops: list[MemoryOp]

SYSTEM = """You maintain a user's long-term memory.
Given EXISTING memories and a NEW exchange, return operations.
Only store durable facts (preferences, constraints, identities, decisions).
Never store secrets, one-off requests, or small talk.
If a new fact contradicts an existing one, UPDATE or DELETE it."""

def plan_memory_ops(existing: list[dict], exchange: str) -> MemoryPlan:
    resp = client.beta.chat.completions.parse(
        model="gpt-4o-mini",  # check current model names in the provider docs
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"EXISTING:\n{existing}\n\nNEW:\n{exchange}"},
        ],
        response_format=MemoryPlan,
    )
    return resp.choices[0].message.parsed

Swap in whichever provider and structured-output API you use. The shape matters more than the vendor.

The read path: retrieve narrowly, inject cheaply

At the start of each turn, retrieve a handful of memories and inject them as a clearly labelled block. Do not stuff 50 memories in "just in case".

import sqlite3, json, math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a)); nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb + 1e-9)

def recall(db, user_id: str, query_vec: list[float], k: int = 5, min_score: float = 0.35):
    rows = db.execute(
        "SELECT id, text, embedding, updated_at FROM memories "
        "WHERE user_id = ? AND deleted = 0", (user_id,)
    ).fetchall()
    scored = [
        (cosine(query_vec, json.loads(emb)), mid, text, ts)
        for mid, text, emb, ts in rows
    ]
    scored = [s for s in scored if s[0] >= min_score]
    return sorted(scored, reverse=True)[:k]

def render(memories) -> str:
    lines = [f"- {text} (as of {ts[:10]})" for _, _, text, ts in memories]
    return "Known about this user (may be outdated; verify if critical):\n" + "\n".join(lines)

Two details that pay off: a minimum score so irrelevant memories are not injected, and timestamps in the rendered block so the model can weigh staleness.

When plain vectors are not enough: time

"Where does the user live?" has a different answer depending on when you ask. A flat vector store returns both the old and new address and lets the model guess. Temporal approaches attach validity windows to facts so superseded facts are closed out rather than deleted. Zep's Graphiti engine is built around this idea of a temporal knowledge graph.

You can get most of the benefit cheaply: add valid_from and valid_to columns, and when an UPDATE arrives, set valid_to on the old row instead of overwriting it. Retrieval filters on valid_to IS NULL by default, and your audit trail comes for free.

Picking a framework

Option	Shape	Strong when	Watch out for
Mem0	Memory layer you add to an existing agent	You want ADD/UPDATE reconciliation fast, framework-agnostic	Benchmark claims vary widely between sources
Zep	Temporal knowledge graph of facts	Time-aware reasoning, compliance needs	More moving parts than a vector table
Letta (ex-MemGPT)	Full stateful agent runtime with core, recall, archival tiers	You want the agent to manage its own memory	You adopt the runtime, not just a library
Roll your own (SQLite/Postgres + pgvector)	Tables + the two functions above	Small scope, strict data residency	You own extraction quality and evals

Published benchmark numbers for these tools disagree depending on who ran them, and many comparison articles are vendor-aligned. Pricing changes often. Run a test on your own conversations and check each vendor's current docs before committing.

Common pitfalls

1. Memory poisoning. Anything the model writes to memory can later be injected into prompts. If a tool result or web page can influence the extractor, an attacker can plant instructions that persist across sessions. Extract only from user and assistant turns you trust, and treat retrieved memory as data, not instructions.
2. No deletion path. Users will ask you to forget things, and regulations may require it. Give every memory an ID, a user scope, and a hard-delete route.
3. Cross-user leakage. Always filter by user or tenant at the query level, not in application code after retrieval. See our post on permission-aware RAG for the pattern.
4. Storing everything. Cost and noise grow together. If the extractor says NOOP most of the time, it is working.
5. Never evaluating recall. Build a small set of conversations with known facts and check: was the right memory retrieved, and was the stale one suppressed?
6. Memory that overrides the user. Present memories as "may be outdated". If the user states something now, current input wins.

A minimal evaluation checklist

• A fact stated in session 1 is recalled correctly in session 3
• A fact corrected in session 2 is not returned in its old form
• Irrelevant memories are not injected into unrelated queries
• Deleting a memory removes it from retrieval immediately
• User A's memories never appear in user B's context
• Token overhead of the memory block stays under your budget

Next steps

Start with the roll-your-own version above on one user-facing workflow. Log every memory operation, review a week of ADDs and UPDATEs by hand, and only then decide whether a managed layer earns its complexity. If you want to build and evaluate agents like this with feedback from practitioners, take a look at our AI engineering cohort, or the PM cohort if you are specifying these features rather than coding them.