Context Engineering: The Skill That Separates Working LLM Apps from Broken Ones
Prompt engineering gets the headlines. Context engineering is what makes production LLM systems actually reliable. Here's the mental model and the code.
Everyone learns prompt engineering first. Write a good system prompt, add some examples, ship it. Then the complaints start: the model ignores instructions, hallucinates details it should have known, answers the wrong question, loses track of what happened three turns ago.
The root cause, nearly every time, is not the prompt. It's what surrounds the prompt — the context.
Context engineering is the discipline of deciding what information reaches the model at inference time, in what form, and in what order. It's the difference between an LLM that consistently does the right thing and one that's unreliable at 8% of requests in ways you can't predict.
This post covers the mental model, the common patterns with working code, and the pitfalls that will bite you if you skip it.
Why context is the real lever
Transformer models don't have memory between calls. Every time you call the API, the model sees exactly what's in the context window — nothing more. Everything it "knows" about your user, your conversation history, your data, and its own instructions comes from what you put in that window.
This means you're always making tradeoffs:
- Relevance vs. completeness: more context isn't always better — irrelevant content dilutes attention
- Cost vs. quality: longer contexts cost more tokens and can be slower
- Recency bias: models pay more attention to content near the end of the context; important instructions buried in the middle get ignored
- Instruction collision: system prompt says one thing, retrieved documents imply another — the model has to choose
A prompt engineer writes good instructions. A context engineer decides what information to include, exclude, compress, and position — so the model is set up to succeed before it reads a single instruction.
The four layers of context
Think of the context window as a structured document with four distinct layers, each with its own job:
| Layer | What goes here | Example |
|---|---|---|
| System | Standing instructions, persona, constraints, output format | "You are a support agent for Acme. Always respond in JSON." |
| Background | Long-lived facts about the user or session | User tier, account history, feature flags |
| Retrieved | Just-in-time knowledge from external sources | RAG results, tool call outputs, API responses |
| Conversation | Recent turns of dialogue | Last N messages |
Most teams only think about the System layer. The Retrieved and Conversation layers are where reliability breaks down.
Pattern 1: Selective retrieval, not full document dump
The most common mistake is passing entire documents into context when only a paragraph is relevant. The model is not a search engine — it can't reliably extract the needle from a 20-page haystack.
Bad:
# Don't do this
with open("product_manual.txt") as f:
full_doc = f.read() # 15,000 tokens
messages = [
{"role": "system", "content": "Answer the user's question."},
{"role": "user", "content": f"Context:\n{full_doc}\n\nQuestion: {user_question}"}
]
Better — chunk and retrieve only what's relevant:
import anthropic
from sentence_transformers import SentenceTransformer
import numpy as np
client = anthropic.Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")
def retrieve_relevant_chunks(query: str, chunks: list[str], top_k: int = 3) -> list[str]:
query_embedding = embedder.encode([query])
chunk_embeddings = embedder.encode(chunks)
scores = np.dot(chunk_embeddings, query_embedding.T).flatten()
top_indices = scores.argsort()[-top_k:][::-1]
return [chunks[i] for i in top_indices]
def answer_with_rag(question: str, all_chunks: list[str]) -> str:
relevant = retrieve_relevant_chunks(question, all_chunks, top_k=3)
context = "\n\n---\n\n".join(relevant)
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
system="Answer the question using only the provided context. If the context doesn't contain enough information, say so.",
messages=[
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}"
}
]
)
return response.content[0].text
Three focused chunks at ~300 tokens each beats one 15,000-token dump almost every time. Check the official Anthropic docs for the latest model IDs and context window limits before using this in production.
Pattern 2: Compress conversation history, don't truncate it
The naive approach to long conversations is to drop old messages when you hit the token limit. This causes the model to forget user preferences stated early in the session.
A better approach is rolling summarization: compress old turns into a summary and keep recent turns verbatim.
def compress_history(
messages: list[dict],
keep_recent: int = 6,
model: str = "claude-haiku-4-5-20251001"
) -> list[dict]:
"""
Summarize all but the last `keep_recent` messages.
Returns a new message list with a summary prepended.
"""
if len(messages) <= keep_recent:
return messages
to_compress = messages[:-keep_recent]
recent = messages[-keep_recent:]
summary_response = client.messages.create(
model=model,
max_tokens=512,
system="Summarize the following conversation in 3-5 bullet points. Focus on facts stated by the user (name, preferences, goals, constraints). Do not include pleasantries.",
messages=[{"role": "user", "content": str(to_compress)}]
)
summary = summary_response.content[0].text
summary_message = {
"role": "user",
"content": f"[Earlier conversation summary]\n{summary}\n[End of summary]"
}
return [summary_message] + recent
Use claude-haiku-4-5-20251001 for the summarization step — it's cheap and fast. Reserve your main model for the actual response.
Pattern 3: Position matters — put instructions last
Studies on LLM attention show models weigh content near the beginning and end of context more heavily than content in the middle ("lost in the middle" effect). Use this deliberately.
Effective ordering for a RAG context:
[SYSTEM PROMPT — short, clear instructions]
[USER BACKGROUND — 3-5 facts about this user]
[RETRIEVED DOCUMENTS — bulk of the context]
[CONVERSATION HISTORY — last few turns]
[FINAL INSTRUCTION — what to do with all of the above] ← repeat your key constraint here
def build_context(
system_instructions: str,
user_background: str,
retrieved_docs: list[str],
history: list[dict],
final_reminder: str
) -> tuple[str, list[dict]]:
doc_block = "\n\n".join(f"[Doc {i+1}]\n{doc}" for i, doc in enumerate(retrieved_docs))
system = f"""{system_instructions}
User context:
{user_background}"""
messages = history + [
{
"role": "user",
"content": f"Relevant information:\n{doc_block}\n\n{final_reminder}"
}
]
return system, messages
The final_reminder might be: "Based on the above, answer the user's last question. Do not reference information not present in the docs." Repeating the key constraint at the end dramatically reduces instruction-following failures.
Pattern 4: Structured context, not prose dumps
The model reads your context like a reader, not a database. Structure helps it find things.
Unstructured (hard for the model):
John is a premium user who signed up in 2024. He's been having issues with the export feature.
He mentioned he uses the API. His account ID is 8821. He prefers concise answers.
Structured (easier to reason about):
<user>
<account_id>8821</account_id>
<tier>premium</tier>
<signup_year>2024</signup_year>
<recent_issue>export feature</recent_issue>
<api_user>true</api_user>
<preference>concise responses</preference>
</user>
XML tags work well here — Claude in particular was trained to pay close attention to content inside tags. JSON works too, but XML is more readable in mixed prose+data contexts.
Pitfalls to avoid
| Pitfall | What goes wrong | Fix |
|---|---|---|
| Kitchen-sink context | You include everything "just in case" — 40k tokens of tangentially related docs | Score and filter retrieval; aim for 3-5 highly relevant chunks |
| Stale instructions | System prompt says "today is March 2025" but it's not | Inject the current date dynamically at call time |
| Instruction collision | Retrieved doc says "do X", system prompt says "never do X" | Add a conflict resolution instruction: "If sources conflict with these guidelines, follow these guidelines." |
| Hard truncation | You slice messages[-10:] and lose the user's stated goal |
Use rolling summarization (Pattern 2) |
| Format ambiguity | You ask for JSON but don't show an example | Always include a schema or example output in the system prompt when asking for structured output |
| Tool output bloat | A tool returns 5,000 tokens; you pass it raw | Summarize tool outputs before injecting into context, or use structured extraction |
How to know if your context is the problem
Run this diagnostic: take a failing example and give the model the perfect context manually (copy-paste exactly the right document, trim everything else). Does the failure go away? If yes, your retrieval or context assembly is broken, not your prompt.
A simple eval harness:
def evaluate_context_quality(
question: str,
context: str,
expected_answer: str,
model: str = "claude-sonnet-4-6"
) -> dict:
response = client.messages.create(
model=model,
max_tokens=512,
system="Answer the question using only the provided context.",
messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}]
)
actual = response.content[0].text
# Use a judge model to evaluate
judge = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
system="Reply with PASS or FAIL only. PASS if the actual answer correctly addresses the expected answer, FAIL otherwise.",
messages=[{"role": "user", "content": f"Expected: {expected_answer}\nActual: {actual}"}]
)
return {
"question": question,
"actual": actual,
"result": judge.content[0].text.strip(),
"tokens_used": len(context.split()) # rough estimate
}
Build a 20-question eval set, run it with different retrieval strategies (top-3 vs. top-5, chunk size 200 vs. 500 tokens), and look at your pass rate. That's the feedback loop that makes context engineering a real engineering practice.
Context engineering vs. prompt engineering — the difference
They're complementary, not competing:
| Prompt engineering | Context engineering | |
|---|---|---|
| Focus | What you tell the model to do | What information the model has to work with |
| Main lever | System prompt, instructions, few-shot examples | Retrieval, history management, document structure |
| Breaks when | Instructions are ambiguous or contradictory | Retrieved content is noisy, stale, or incomplete |
| Measured by | Instruction-following rate | Answer correctness, grounding rate, hallucination rate |
A well-crafted prompt on a poorly constructed context fails. A mediocre prompt on carefully curated context often succeeds. Fix context first.
Where to go next
The techniques above — selective retrieval, rolling summarization, position-aware assembly, and structured formatting — handle 80% of reliability problems in production LLM apps. The remaining 20% usually comes down to eval gaps (you don't know what's failing) or retrieval quality (what you're pulling back isn't actually relevant).
If you're building production AI systems and want to go deeper on context engineering, retrieval pipelines, and systematic evals, the Joinloop AI Engineering cohort covers all of this hands-on — from chunking strategy and embedding models to running structured eval suites against real applications.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.