AI Agent Memory: The Four Layers Every Production Agent Needs

Agents fail in production for surprisingly mundane reasons. Not hallucinations. Not reasoning errors. They fail because they forget.

A customer support agent forgets what the user said three turns ago. A coding agent re-implements a helper it wrote last week. A research agent fetches the same URL it already processed. Every one of these is a memory problem — and most agent frameworks treat memory as an afterthought.

This post lays out the four-layer memory model that mirrors how human cognition works, shows you where each layer belongs in your stack, and gives you code you can run today.

Why the single-context approach breaks

The naive approach is to stuff everything into one big system prompt: conversation history, user facts, past actions, background knowledge. This works until it doesn't.

Problems that appear at scale:

• Context limits: Even with 200k-token windows, long-running agents hit the ceiling. More importantly, stuffing the context window costs money per call.
• Relevance decay: Older turns get less attention (the "lost in the middle" problem). The model skims things buried 80k tokens deep.
• No persistence across sessions: Clear the context, lose everything. Users hate reintroducing themselves.
• No shared state across agent instances: If you run 10 parallel agents on the same task, they each reinvent the wheel.

The solution is to treat memory as a first-class system with distinct layers, each with different storage, retrieval, and update characteristics.

The four memory layers

Layer	Analogous to	Where stored	Lifespan	Updated by
Working memory	RAM	LLM context window	Per turn	Automatically (context accumulation)
Episodic memory	Diary	Vector DB / key-value store	Per session or forever	Agent writes at checkpoints
Semantic memory	Encyclopedia	Vector DB / knowledge graph	Long-lived	Offline pipeline or admin
Procedural memory	Muscle memory	Prompt templates / tool definitions	Stable	Human engineer

Each layer has a different job. Mixing them up — or ignoring three of them — is what causes most agent amnesia bugs.

---

Layer 1: Working memory (in-context)

This is your context window. It holds the current conversation, tool call results, and whatever you've explicitly injected. Nothing special to implement — it's just the messages array you pass to the model.

The key discipline: be deliberate about what you put in it.

def build_working_memory(
    conversation_history: list[dict],
    retrieved_episodic: str,
    retrieved_semantic: str,
    max_history_turns: int = 10,
) -> list[dict]:
    messages = []

    # Inject retrieved long-term memory as a system message
    if retrieved_episodic or retrieved_semantic:
        context_block = ""
        if retrieved_semantic:
            context_block += f"## Background knowledge\n{retrieved_semantic}\n\n"
        if retrieved_episodic:
            context_block += f"## What happened before\n{retrieved_episodic}\n\n"

        messages.append({
            "role": "system",
            "content": f"Use this context to inform your response:\n\n{context_block}",
        })

    # Trim conversation history to last N turns to control token usage
    trimmed = conversation_history[-max_history_turns * 2:]
    messages.extend(trimmed)

    return messages

The critical insight: don't dump raw episodic/semantic memory into the conversation history. Put it in a dedicated system message with clear headers. The model attends to it better, and you can update it independently of the conversation.

---

Layer 2: Episodic memory

Episodic memory stores what happened in past interactions: user preferences stated explicitly, decisions the agent made, errors encountered, task outcomes.

Think of it as a compressed diary the agent can query before responding.

Storage

A vector database (pgvector, Qdrant, Weaviate) works well because retrieval is semantic — you want memories relevant to the current situation, not just the most recent ones.

from openai import OpenAI
import psycopg2
import json
from datetime import datetime

client = OpenAI()

def embed(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return response.data[0].embedding

def save_episode(conn, user_id: str, content: str, metadata: dict = None):
    """Write a memory episode after a significant agent action."""
    embedding = embed(content)
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO episodes (user_id, content, embedding, metadata, created_at)
            VALUES (%s, %s, %s, %s, %s)
            """,
            (user_id, content, embedding, json.dumps(metadata or {}), datetime.utcnow()),
        )
    conn.commit()

def retrieve_episodes(conn, user_id: str, query: str, top_k: int = 5) -> list[str]:
    """Retrieve relevant past episodes for the current context."""
    query_embedding = embed(query)
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT content
            FROM episodes
            WHERE user_id = %s
            ORDER BY embedding <-> %s::vector
            LIMIT %s
            """,
            (user_id, query_embedding, top_k),
        )
        rows = cur.fetchall()
    return [row[0] for row in rows]

When to write episodes

Don't write after every turn — you'll create noise. Write at semantic checkpoints:

• Task completed (success or failure)
• User stated a preference ("I prefer TypeScript over Python")
• Agent made a consequential decision
• Error encountered and resolved

def should_save_episode(agent_response: str, tool_calls: list) -> bool:
    """Heuristic: save if task concluded or preference was expressed."""
    conclusive_phrases = ["completed", "finished", "done", "failed", "error", "prefer", "always", "never"]
    return (
        any(p in agent_response.lower() for p in conclusive_phrases)
        or any(tc["function"]["name"] in ("mark_complete", "report_error") for tc in tool_calls)
    )

---

Layer 3: Semantic memory

Semantic memory is your knowledge base — facts about the world that are stable and not tied to any particular interaction. Product documentation, user profile data, API schemas, domain knowledge.

This is where RAG lives. The difference from episodic memory is that semantic memory is curated, not written by the agent itself.

def retrieve_semantic(conn, query: str, namespace: str = "docs", top_k: int = 3) -> str:
    """Retrieve relevant knowledge base chunks."""
    query_embedding = embed(query)
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT content, source_url
            FROM knowledge_chunks
            WHERE namespace = %s
            ORDER BY embedding <-> %s::vector
            LIMIT %s
            """,
            (namespace, query_embedding, top_k),
        )
        rows = cur.fetchall()

    if not rows:
        return ""

    chunks = []
    for content, source in rows:
        chunks.append(f"Source: {source}\n{content}")

    return "\n\n---\n\n".join(chunks)

Semantic vs. episodic: the practical distinction

The rule of thumb: if it came from the agent's experience, it's episodic. If it was authored by a human, it's semantic.

User says "I prefer short responses" → episodic (agent learned this from interaction).
Product spec for a feature → semantic (was written by a PM and indexed).

Don't merge these into one retrieval pool — they have different update cadences and different levels of authority. A product spec should override an agent's inferred preference.

---

Layer 4: Procedural memory

Procedural memory is your tool definitions and system prompts — the hardwired knowledge of how to do things. This is the most overlooked layer because it's not "memory" in the obvious sense.

But consider: when a human learns to ride a bike, that knowledge isn't stored as explicit facts — it's encoded in patterns and reflexes. Similarly, the agent's ability to call tools correctly, format outputs, and follow conventions lives in procedural memory.

Practical manifestations:

• Tool schemas: The definition of search_web(query: str) -> list[SearchResult] is procedural memory.
• System prompt patterns: Step-by-step instructions for how the agent should approach a task class.
• Few-shot examples: Embedded examples that teach format and reasoning style.

PROCEDURAL_MEMORY = {
    "code_review": """
When reviewing code:
1. Check for security issues first (injections, exposed secrets, auth gaps)
2. Then correctness (logic errors, edge cases)
3. Then style (naming, structure)
Always cite the specific line number. Never praise unnecessarily.
""",
    "research_task": """
When researching a topic:
1. Identify the specific claim to verify
2. Find at least 2 independent sources
3. Note any contradictions between sources
4. Summarise with confidence level: High / Medium / Low
""",
}

def get_procedure(task_type: str) -> str:
    return PROCEDURAL_MEMORY.get(task_type, "")

Procedural memory is the only layer that humans (not the agent) update. Keep it in version control, treat changes carefully — it's the agent's training data at inference time.

---

Putting it together: the memory retrieval pipeline

A complete agent turn with all four layers:

async def agent_turn(
    user_message: str,
    user_id: str,
    session_history: list[dict],
    task_type: str,
    db_conn,
) -> str:
    # 1. Retrieve long-term memories relevant to the current message
    episodes = retrieve_episodes(db_conn, user_id, query=user_message)
    episodic_context = "\n".join(f"- {e}" for e in episodes)

    semantic_context = retrieve_semantic(db_conn, query=user_message)

    # 2. Build working memory from all layers
    messages = build_working_memory(
        conversation_history=session_history,
        retrieved_episodic=episodic_context,
        retrieved_semantic=semantic_context,
    )

    # 3. Prepend procedural memory as the primary system prompt
    procedure = get_procedure(task_type)
    messages.insert(0, {"role": "system", "content": procedure})

    # 4. Add the current user message
    messages.append({"role": "user", "content": user_message})

    # 5. Call the model
    response = client.chat.completions.create(
        model="claude-sonnet-5",  # or your preferred model
        messages=messages,
    )
    reply = response.choices[0].message.content

    # 6. Decide whether to persist an episode
    if should_save_episode(reply, []):
        save_episode(db_conn, user_id, content=f"User asked: {user_message}\nAgent did: {reply[:300]}")

    # 7. Update session history (working memory for next turn)
    session_history.append({"role": "user", "content": user_message})
    session_history.append({"role": "assistant", "content": reply})

    return reply

---

Common pitfalls

1. Writing episodes too frequently
If you write an episode after every turn, retrieval becomes noisy. The vector search returns stale, redundant chunks. Write at semantic checkpoints, not on a fixed schedule.

2. Forgetting to scope by user
Always filter episodic retrieval by user_id or session_id. Without scoping, one user's preferences bleed into another's context. This is both a product bug and a potential privacy issue.

3. Treating semantic memory as static
Knowledge goes stale. If your product changes and the semantic store still has old documentation, the agent confidently gives wrong answers. Build a pipeline to re-index semantic memory when source documents change.

4. Putting procedural memory in episodic storage
Don't let the agent rewrite its own procedures at runtime based on user feedback. "The user told me to always respond in bullet points" should live in episodic memory and be soft guidance — it shouldn't overwrite the system prompt.

5. Context injection order matters
Most models attend more strongly to content near the beginning and end of the context. Put your most authoritative content (procedural memory) first, retrieved context second, conversation last. Don't bury the knowledge base in the middle.

---

Choosing your storage backend

Backend	Best for	Hosted option	Self-hosted
pgvector	Teams already on Postgres; unified SQL + vector	Supabase, Neon	Postgres + extension
Qdrant	High-throughput retrieval; rich filtering	Qdrant Cloud	Docker
Redis	Session-scoped working memory; low latency	Redis Cloud	Redis Stack
SQLite + custom embedding	Local dev, prototyping	—	Built-in

For most production systems starting out: Postgres + pgvector for both episodic and semantic memory, Redis for session state. Add a dedicated vector DB only when you hit retrieval latency constraints.

---

What to build next

If you have an agent running in production today:

1. Add episodic memory first — it gives the biggest user experience improvement with the least complexity.
2. Instrument what gets written and retrieved. Log every save_episode and retrieve_episodes call. You'll quickly see what's noise and what's valuable.
3. Audit your procedural memory (system prompt). Is it doing too much? Split task-specific procedures into per-task-type strings rather than one monolithic prompt.
4. Build semantic memory last, and only if you have a stable corpus. Maintaining a knowledge base requires a real pipeline — don't half-build it.

The agents that feel intelligent aren't necessarily smarter at reasoning. They remember better.

---

If you want to go deeper on building production-grade AI agents — architecture, evaluation, deployment — check out the Joinloop AI Engineering cohort. It's a practitioner-first program built for engineers who want to ship real systems, not just run notebooks.