Headroom: Cut Your LLM Token Usage by 60–95% Without Changing Your Answers

If you've built a real AI agent — not a demo, but one that calls tools, reads logs, fetches search results, and pulls code files — you've hit the token wall. A single agentic loop that reads a directory, queries a database, and fetches a GitHub issue can easily burn 40,000–70,000 tokens per turn. At frontier model pricing, that adds up fast. And it's not just cost: longer contexts mean slower responses, degraded reasoning at the boundaries, and harder-to-hit KV cache hits.

Headroom is a new open-source library (Apache 2.0) that tackles this directly. It sits between your agent and the LLM, compresses what it sees before passing it through, and maintains a local store of originals so the LLM can retrieve them if needed. The headline claim: 60–95% fewer tokens, same answers.

Let's look at what it actually does, how to use it, and where it falls short.

The Token Bloat Problem

A typical agent loop does something like:

1. Call a search tool → returns 50 JSON results
2. Read 3 source files → 800 lines of code total
3. Query a database → structured rows dump
4. Pass all of this into a single context window for the LLM to reason over

Each of those tool results comes back as raw text. JSON dumps include schema boilerplate. Code files include imports, comments, and dead code. Log files repeat the same pattern hundreds of times. By the time you've aggregated a few turns of a complex task, your context is enormous — and most of it is noise.

The naive fix is chunking or summarizing, but both lose information in ways that hurt agent accuracy. What you actually want is something that understands the content type and compresses intelligently — keeping what matters, discarding what doesn't.

What Headroom Does

Headroom adds a compression stage before messages reach your LLM. Its two-stage pipeline works like this:

Stage 1 — CacheAligner: Stabilizes message prefixes so repeated system prompt content hits your LLM provider's KV cache. This alone can cut latency on repeated calls.

Stage 2 — ContentRouter: Detects the content type of each message and routes it to the appropriate compressor. The router handles:

Content Type	Algorithm	What It Preserves
JSON / structured data	SmartCrusher	Anomalies, outliers, schema shape
Source code	CodeCompressor (AST-aware)	Function signatures, change hunks
Plain text / prose	Kompress (ModernBERT)	Semantically dense tokens
Logs	LogCompressor	Errors, failures, stack traces
Search results	SearchCompressor	Ranked relevance
Git diffs	DiffCompressor	Change hunks, context lines
HTML pages	HTMLExtractor	Article text, removes markup

Critically, originals are never discarded. Compressed content goes into a CCR (Compress-Cache-Retrieve) store — a local store that gives the LLM a retrieval tool to fetch full originals on demand. The LLM sees compressed summaries by default, but can ask for the original when it needs to reason about exact content.

Three Ways to Integrate

Headroom supports three integration modes depending on how much you want to change your code.

Option 1: Proxy (zero code changes)

pip install "headroom-ai[all]"
headroom proxy

Then point your SDK at the local proxy:

# Anthropic
ANTHROPIC_BASE_URL=http://localhost:8787 python your_agent.py

# OpenAI
OPENAI_BASE_URL=http://localhost:8787/openai python your_agent.py

The proxy intercepts every request, compresses the messages, and forwards to the real API. Your code is unchanged. This is the fastest way to evaluate whether Headroom helps your specific workload.

Option 2: Python Library

For tighter control — especially if you want to inspect compression results or tune parameters:

from anthropic import Anthropic
from headroom import compress

client = Anthropic()

messages = [
    {
        "role": "user",
        "content": "Here's the log output from the last deployment:\n" + huge_log_blob
    }
]

# Compress before sending
result = compress(messages, model="claude-sonnet-4-6")

print(f"Tokens before: {result.original_tokens}")
print(f"Tokens after:  {result.compressed_tokens}")
print(f"Reduction:     {result.reduction_pct:.0f}%")

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    messages=result.messages,
    tools=result.tools,   # Includes CCR retrieval tool if originals were stored
)

The result.tools field is important: Headroom injects a retrieval tool that lets the LLM call back for originals when it needs them. You pass these alongside your own tool list.

Option 3: TypeScript / Node.js

import Anthropic from "@anthropic-ai/sdk";
import { compress } from "headroom-ai";

const client = new Anthropic();

const messages = [
  {
    role: "user" as const,
    content: `Search results:\n${jsonSearchResults}`,
  },
];

const result = await compress(messages, { model: "claude-sonnet-4-6" });

const response = await client.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 4096,
  messages: result.messages,
  tools: result.tools,
});

LangChain Integration

If you're using LangChain, Headroom wraps at the model layer:

from langchain_openai import ChatOpenAI
from headroom.integrations import HeadroomChatModel

base_llm = ChatOpenAI(model="gpt-4o")
llm = HeadroomChatModel(base_llm)

# Use llm exactly as you would ChatOpenAI — compression is transparent
chain = prompt | llm | output_parser
result = chain.invoke({"context": huge_document, "question": "What failed?"})

Real-World Benchmarks

These are from Headroom's documented production examples:

Workload	Tokens Before	Tokens After	Savings
Code search (100 results)	17,765	1,408	92%
SRE incident debugging	65,694	5,118	92%
GitHub issue triage	54,174	14,761	73%
Typical agentic loop	varies	varies	60–87% avg

Accuracy benchmarks (GSM8K math, TruthfulQA, SQuAD v2, BFCL tool-use) held steady post-compression across these workloads.

Common Pitfalls

1. Expecting 90%+ on prose-heavy content The big savings (80%+) come from structured content: JSON, code, logs, HTML. On dense prose — think long PDF chapters or free-text customer feedback — Kompress (the text compressor) delivers more modest gains, typically 30–50%. Set your expectations by content type.

2. Skipping the CCR retrieval tool If you pass result.tools to your LLM call, the model can retrieve originals when it needs to reason about exact values. If you don't pass it, the model only sees the compressed version and may hallucinate details that were trimmed. Always pass result.tools alongside your own tools.

3. Compressing messages the LLM already produced Only compress incoming context (tool results, retrieved documents, external data). Don't run compression over your system prompt or the model's prior responses — these are already dense and compression will degrade them.

4. Ignoring cache alignment The CacheAligner stage is doing real work: it stabilizes the prefix so your LLM provider's KV cache gets consistent hits. If you reconstruct your system prompt dynamically (injecting timestamps, request IDs, etc.), the cache never hits. Move dynamic content out of the system prompt prefix.

5. Not measuring your actual workload first Run the proxy for a week on real traffic and look at the per-request compression stats before optimizing further. Some workloads benefit enormously; others barely move. Don't optimize blind.

Deployment Decision Matrix

You want	Use
Zero code changes, fast evaluation	Proxy
Per-request control + observability	Python / TS library
Claude / Cursor tool compatibility	MCP server (headroom mcp)
LangChain agent with transparent compression	HeadroomChatModel wrapper
100+ providers including Bedrock, Vertex	Proxy via LiteLLM

Is It Worth It?

At small scale (a few hundred requests a day), the savings may not matter much. But if you're running:

• Long agentic loops with many tool calls per turn
• RAG pipelines where retrieved chunks dominate the context
• SRE / DevOps agents that read logs and diffs
• Code review agents that process large diffs or codebases

...then Headroom is worth the 10-minute integration cost. The proxy path literally requires two commands and one environment variable.

For teams running agents at scale, the math is compelling: if you're spending $5,000/month on tokens for an internal agent system, a conservative 60% reduction saves $3,000/month. The tool is Apache 2.0 and self-hosted, so there's no additional cost.

The CCR retrieval design also solves a real problem that naive truncation doesn't: the LLM retains access to full originals when it needs them, rather than being forced to reason from an incomplete summary.

Next Steps

• GitHub repo — start with pip install "headroom-ai[all]" && headroom proxy
• Check result.reduction_pct on your actual workload before committing to library integration
• If you use MCP-based tools in Claude or Cursor, run headroom mcp to get compression at the tool output layer

---

If you're building production AI agents and want to understand the full stack — from context management to evals to cost optimization — that's exactly what we cover in Joinloop's AI Engineering cohort. Practical, code-first, and built for engineers shipping real systems.