Context Compaction for Long-Running Agents: Summarize, Trim, or Checkpoint

An agent that handles a ten-turn support chat is easy. An agent that runs a two-hour refactor, reads forty files, and calls thirty tools is a different system. Somewhere around turn forty the context window fills up, and one of three things happens. The provider rejects the request. The oldest messages silently fall off and the agent forgets the constraint you gave it at the start. Or the model keeps working, but the early decisions are buried so deep that it contradicts them.

The fix is not "use a bigger context window." Bigger windows cost more per call, get slower, and still degrade on details buried in the middle. The fix is to decide, deliberately, what the agent keeps verbatim, what it compresses, and what it writes to durable storage so it can survive a crash. That is context compaction.

This post covers the three strategies, when each one fits, and a runnable compactor you can drop into a Python agent loop.

The problem: not all context is equal

Long-running agent context usually contains four kinds of material with very different lifespans:

1. Instructions. The system prompt and task definition. These must never be compacted. If you lose them, the agent drifts.
2. Decisions and constraints. "Use the v2 API, do not touch the billing module, the customer ID is acct_8812." Short, high-value, and easy to lose in a summary if you are careless.
3. Working detail. Tool outputs, file contents, search results. Large, useful for the next step, and almost worthless ten steps later.
4. Recent turns. The last few exchanges, which the model needs verbatim to continue coherently.

A naive "keep the last N messages" policy treats all four the same. It drops decisions from turn 3 while keeping a 40KB file dump from turn 38. Compaction is about treating these categories differently.

The three strategies

Trim. Drop or truncate old material. Cheap, deterministic, and lossy. Good for tool outputs you can re-fetch, such as a file read that can be repeated. Bad for anything the agent cannot recover.

Summarize. Ask a model to compress older turns into a running summary that stays in context. This preserves meaning at the cost of one extra LLM call per compaction and some fidelity loss. The risk is that the summarizer drops a detail that matters, such as an exact ID or a negative constraint ("do not", "never").

Checkpoint. Write the agent's state (messages, summary, scratch notes, pending tasks) to durable storage on every step. This does not reduce the context size by itself. It makes the run resumable. If the process dies at turn 60, you restart from turn 60 instead of turn 0. Checkpointing is what turns a context problem into an ops problem you already know how to solve.

They compose. A production agent usually trims tool output aggressively, summarizes older conversation when it crosses a budget, and checkpoints every step.

The compactor

The design below does three things. It keeps the last keep_recent messages verbatim. When the total exceeds a token budget, it folds everything older into a single running summary, and it writes state to disk after every message via an atomic rename so a crash never leaves a half-written file.

The system prompt is not stored in the compactor. You pass it separately on every call, so it is never summarized away.

import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable

Message = dict[str, str]  # {"role": "user" | "assistant" | "tool", "content": "..."}


def estimate_tokens(messages: list[Message]) -> int:
    # Rough heuristic: about 4 characters per token for English text.
    # Swap in your provider's tokenizer if you need exact counts.
    return sum(len(m["content"]) for m in messages) // 4


@dataclass
class Compactor:
    summarize: Callable[[str], str]  # any LLM call: prompt in, text out
    budget_tokens: int = 8000
    keep_recent: int = 6
    checkpoint_path: Path | None = None
    summary: str = ""
    messages: list[Message] = field(default_factory=list)

    def add(self, role: str, content: str) -> None:
        self.messages.append({"role": role, "content": content})
        if estimate_tokens(self.context()) > self.budget_tokens:
            self._compact()
        self._checkpoint()

    def context(self) -> list[Message]:
        """The non-system history to send to the model, after the system prompt."""
        head: list[Message] = []
        if self.summary:
            head = [{"role": "user", "content": f"Summary of earlier conversation:\n{self.summary}"}]
        return head + self.messages

    def _compact(self) -> None:
        if len(self.messages) <= self.keep_recent:
            return  # nothing old enough to fold away
        old = self.messages[: -self.keep_recent]
        self.messages = self.messages[-self.keep_recent :]
        transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old)
        prior = f"Existing summary:\n{self.summary}\n\n" if self.summary else ""
        self.summary = self.summarize(
            "Update the summary of this conversation so the assistant can continue the task. "
            "Preserve decisions, explicit constraints (including 'do not' rules), open questions, "
            "and exact identifiers such as IDs, paths, and versions. Be concise.\n\n"
            + prior
            + "New messages:\n"
            + transcript
        )

    def _checkpoint(self) -> None:
        if self.checkpoint_path is None:
            return
        state = {"summary": self.summary, "messages": self.messages}
        tmp = self.checkpoint_path.with_suffix(".tmp")
        tmp.write_text(json.dumps(state, indent=2))
        tmp.replace(self.checkpoint_path)  # atomic rename: no half-written state file

    @classmethod
    def resume(cls, path: Path, summarize: Callable[[str], str], **kwargs) -> "Compactor":
        c = cls(summarize=summarize, checkpoint_path=path, **kwargs)
        if path.exists():
            state = json.loads(path.read_text())
            c.summary = state["summary"]
            c.messages = state["messages"]
        return c

Three details matter here:

• The summary is updated, not appended. Each compaction feeds the previous summary back in, so the summary stays one block of bounded size instead of growing into a stack of summaries.
• Constraints are named explicitly in the summarizer prompt. Generic "summarize this" prompts drop negative rules first. Calling them out is cheap insurance.
• The atomic write. write_text on the final path can leave a truncated file if the process dies mid-write. Writing to a temp file and calling replace is atomic on POSIX filesystems, so you always resume from either the old or the new state.

Here is how you wire it into a loop with a real model client:

from pathlib import Path

def call_llm(prompt: str) -> str:
    # Replace with your provider SDK call. Keep the summarizer on a cheap model.
    ...

SYSTEM = "You are a refactoring agent. Never modify files under billing/."

state_file = Path("runs/refactor-7731.json")
ctx = Compactor.resume(state_file, summarize=call_llm, budget_tokens=8000, keep_recent=6)

ctx.add("user", "Migrate the customer lookup from v1 to v2 API. Customer ID is acct_8812.")
# ... each agent step calls ctx.add(...) with the assistant reply and tool results

The resume call is what makes the run survive restarts. If the process dies, the next launch reloads the summary and recent messages and continues.

Testing the compactor without an API key

You should test compaction the same way you test any other stateful component: with a deterministic fake summarizer. This script checks that the budget is respected, the summary is updated rather than duplicated, and state survives a restart.

import tempfile
from pathlib import Path

calls = []

def fake_summarize(prompt: str) -> str:
    calls.append(prompt)
    return f"summary-v{len(calls)}"

with tempfile.TemporaryDirectory() as d:
    path = Path(d) / "state.json"
    c = Compactor.resume(path, summarize=fake_summarize, budget_tokens=200, keep_recent=4)
    for i in range(30):
        c.add("user" if i % 2 == 0 else "assistant", f"step {i} " + "x" * 80)

    assert len(c.messages) <= 4 + 1, "recent window should stay small"
    assert c.summary == f"summary-v{len(calls)}", "summary should be the latest one"
    assert "Existing summary" in calls[-1] if len(calls) > 1 else True

    restored = Compactor.resume(path, summarize=fake_summarize, budget_tokens=200, keep_recent=4)
    assert restored.summary == c.summary
    assert restored.messages == c.messages
    print("compactor ok:", len(calls), "compactions")

Running this should print a compaction count, and the assertions confirm the three properties that break most often in practice. Assertion failures here are usually an off-by-one in keep_recent, not a bug in the model.

Comparison: which strategy for which material

Material	Strategy	Why
System prompt, task definition	Never compact	Losing it causes silent drift
Decisions, constraints, IDs	Summarize with explicit preservation rules	Must survive, but is small enough to keep as prose
Large tool output (file reads, search results)	Trim, or store externally and keep a reference	Re-fetchable, so storing a pointer beats keeping the bytes
Last few turns	Keep verbatim	The model needs exact wording to continue
Whole run state	Checkpoint every step	Enables resume after crashes and retries

Approach	Cost	Fidelity	Recoverable after crash
Keep last N messages	None	Loses early decisions	No
Trim tool outputs	None	Lossy only for big outputs	No, unless checkpointed
Rolling summary	One LLM call per compaction	Good prose, risks dropping details	Only with checkpoints
Checkpoint every step	Disk writes	No effect on context	Yes

Common pitfalls

Summarizing the summary without bounds. If the summary grows every compaction, you have rebuilt the original problem one level up. Cap the summary length in the prompt and check its token count. If it exceeds a fixed limit, summarize it again into a shorter form, or move stable facts into a separate structured "facts" store.

Losing negative constraints. "Do not modify billing/" is more fragile than "modify billing/". Summarizers tend to convert prohibitions into neutral statements. Put the rules in the system prompt, where they are never compacted, and keep only per-task constraints in the summary.

Compacting in the middle of a tool call. If the assistant issued a tool call and the result has not arrived yet, a compaction that cuts between them breaks the message structure many providers require. Compact only at turn boundaries.

Using the expensive model to summarize. Compaction runs often and the output is internal. A smaller model is usually enough. Measure the summary against a golden set of constraint-recall questions before you switch.

Trusting the token estimate. The four-characters-per-token heuristic drifts badly on code, JSON, and non-English text. Log the provider-reported token usage on every call and recalibrate the budget from real numbers.

Checkpointing secrets. The checkpoint file now contains everything the agent saw, including any credentials that passed through a tool result. Treat it with the same care as a log file: restrict permissions and redact before writing.

Next steps

1. Instrument first. Log message count, estimated tokens, and provider-reported input tokens per call. You cannot tune a budget you cannot see.
2. Trim tool output at the source. Cap each tool result (for example, 4KB with a note saying it was truncated) before it reaches the compactor. This removes the most common cause of runaway context.
3. Build a constraint-recall test. Write five questions whose answers depend on turn-1 constraints, run the agent past several compactions, and check the answers. Run this in CI so a summarizer change cannot silently regress.
4. Add checkpoints before you need them. Retrofitting resumability into a long agent is painful. The compactor above is about 60 lines, so add it on day one.

If you are building agents that run longer than a single chat session, this is the kind of work we cover hands-on in the Joinloop AI Engineering cohort, including building and evaluating these loops against real failure cases.