Harness Engineering: Tune the Agent's Environment Before You Touch the Model

Teams building coding agents, support agents, or any tool-using LLM system tend to reach for the same lever when something goes wrong: swap the model. Try a bigger one, try a different provider, add a fine-tune. It's the most expensive lever and often the wrong one.

The harness — the code around the model that defines its tools, formats its context, retries its failures, and enforces its limits — usually accounts for more of an agent's real-world reliability than the model weights do. Teams doing this well have shown that tuning the harness instead of the model can get a mid-tier model within a point or two of a frontier model's benchmark score, at a fraction of the cost. That's not a fine-tuning result. That's an engineering result.

This post is a practical walkthrough of the harness levers worth pulling, in the order you should pull them.

What "the harness" actually is

If you strip an agent down to its loop, it looks like this:

while not done:
    context = build_context(state)
    response = model.call(context, tools)
    if response.tool_calls:
        results = execute_tools(response.tool_calls)
        state.update(results)
    else:
        done = True

Everything in that loop except model.call is harness code. That includes:

• Tool definitions — names, descriptions, parameter schemas
• Context construction — what goes into the prompt, in what order, how it's truncated
• Execution — how tool calls run, what errors look like, retry and timeout policy
• State management — what persists across turns, what gets summarized or dropped
• Guardrails — validation before and after tool execution

None of this touches model weights. All of it determines whether the agent actually completes the task.

Lever 1: Tool schemas, not tool code

The single highest-leverage change most teams can make is rewriting tool descriptions and parameter schemas. Models pick the wrong tool, or call the right tool with malformed arguments, far more often because of ambiguous schemas than because of reasoning failures.

Compare these two definitions of the same tool:

# Weak — ambiguous, no constraints, no examples
{
    "name": "search",
    "description": "Search for stuff",
    "parameters": {
        "query": {"type": "string"}
    }
}

# Strong — scoped, constrained, self-documenting
{
    "name": "search_internal_docs",
    "description": (
        "Search the internal engineering wiki for design docs, runbooks, "
        "and API references. Does NOT search code, tickets, or Slack. "
        "Use search_codebase for code questions."
    ),
    "parameters": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "2-6 keyword search terms, not a full sentence."
            },
            "max_results": {
                "type": "integer",
                "minimum": 1,
                "maximum": 10,
                "default": 5
            }
        },
        "required": ["query"]
    }
}

The second version does three things the first doesn't: it disambiguates the tool from its neighbors (models confuse similarly-named tools constantly when there are 10+ in scope), it constrains the input shape instead of hoping the model infers it, and it tells the model what not to do with it. When you have more than five or six tools available, this disambiguation matters more than almost anything else in the harness.

If you're debugging an agent that keeps calling the wrong tool, don't reach for a bigger model first — read your tool descriptions out loud and ask whether a new hire with no context would pick correctly.

Lever 2: Context construction

The order and format of what you put in context changes behavior even when the total information is identical. Two harness patterns worth adopting:

Put constraints and current state near the end of the context, closest to the generation point. Models weight recent context more heavily in practice, regardless of official "long context" claims. A system prompt with your rules at the top and 40k tokens of file contents after it means the rules are effectively diluted by the time generation starts. Restate the binding constraints right before the model has to act.

Structure over prose for machine-generated context. If you're injecting tool results, file diffs, or logs into context, use a consistent structured format (XML-ish tags or fenced blocks) rather than natural language summaries. It's both easier for the model to parse and easier for you to truncate deterministically when you're near a context limit.

def build_context(state):
    return f"""
{state.system_prompt}

<files>
{format_files(state.open_files)}
</files>

<recent_tool_results>
{format_tool_results(state.last_n_results, n=3)}
</recent_tool_results>

<current_task>
{state.task}
</current_task>

<constraints>
{state.constraints}
</constraints>
"""

Notice constraints is last. That's deliberate.

Lever 3: Retry and timeout policy

This is the most-skipped harness component and the one that causes the most silent failures in production. A tool call that times out, returns malformed JSON, or hits a transient API error should not just bubble up as a generic error string — the model needs enough signal to decide whether to retry, rephrase, or abandon.

import time

def execute_tool_with_policy(tool_call, max_retries=2, timeout_s=10):
    for attempt in range(max_retries + 1):
        try:
            result = run_tool(tool_call, timeout=timeout_s)
            return {"status": "ok", "result": result}
        except TimeoutError:
            if attempt == max_retries:
                return {
                    "status": "error",
                    "error": "timeout",
                    "detail": f"Tool exceeded {timeout_s}s after {attempt+1} attempts. Consider a narrower query.",
                }
            time.sleep(0.5 * (2 ** attempt))
        except ValidationError as e:
            # Don't retry malformed args — the model needs to fix the call, not repeat it
            return {"status": "error", "error": "invalid_args", "detail": str(e)}
    return {"status": "error", "error": "unknown"}

The key design decision: distinguish errors that are worth retrying automatically (timeouts, rate limits) from errors that need the model to change its next call (bad arguments, tool misuse). Auto-retrying a malformed call just burns tokens repeating the same mistake.

Lever 4: Guardrails at the boundary, not in the prompt

"Please don't delete files outside the working directory" in a system prompt is a suggestion. A path-resolution check before the tool executes is a guarantee. Anything you can enforce in code, enforce in code — reserve prompt-level instruction for judgment calls the harness genuinely can't make deterministically.

def validate_file_write(path: str, allowed_root: str) -> bool:
    resolved = os.path.realpath(path)
    return resolved.startswith(os.path.realpath(allowed_root) + os.sep)

This is a five-line function that eliminates an entire class of agent failure that no amount of prompting reliably prevents.

Comparison: harness levers vs. model levers

Lever	Cost to change	Iteration speed	Fixes
Tool schema rewrite	Low	Minutes	Wrong tool selection, malformed args
Context ordering/format	Low	Minutes	Instruction-following drift over long context
Retry/timeout policy	Low-medium	Hours	Transient failures reported as task failures
Boundary guardrails	Medium	Hours	Unsafe or destructive actions
Prompt rewrite	Low	Minutes	Task framing, tone, edge-case handling
Fine-tuning	High	Days-weeks	Deep domain-specific reasoning gaps
Swap to bigger model	Low effort, high $	Minutes	Nothing structural — just raw capability

Everything above the double line should be exhausted before you reach for the two rows below it. Most "the model isn't smart enough" failures are actually "the harness didn't give the model what it needed" failures.

Common pitfalls

• Blaming the model before reading your own logs. If you can't see the exact context and tool results the model received on a failed run, you're debugging blind. Log full context, not just the final response.
• Too many tools in scope at once. Beyond ~15-20 tools, selection accuracy drops regardless of model size. Scope tools to the current task phase instead of exposing everything always.
• Silent truncation. If you're dropping older messages or file contents to fit a context window, the model has no idea information is missing. Say so explicitly in the context: [... 3 earlier files omitted for length ...].
• Retrying non-idempotent tool calls. A retry policy that re-runs a send_email or create_ticket call on timeout can double-execute the action. Gate retries on idempotency, not just on error type.
• No eval loop for harness changes. If you can't measure whether a schema rewrite improved tool-selection accuracy, you're guessing. Keep a small labeled set of past failures and rerun them after every harness change.

Next steps

Start with an audit, not a rewrite: pull the last 20 failed agent runs from your logs, and for each one, ask whether the failure was a reasoning gap or a harness gap — ambiguous tool schema, missing context, silent truncation, no retry signal. In most production agents, the split runs 70/30 in favor of the harness. Fix those first; they're cheaper, faster to iterate on, and don't require touching the model at all.

If you want structured practice building and debugging agent harnesses — tool design, context engineering, and the eval loops to measure whether your changes actually help — that's core to the curriculum in Joinloop's AI engineering cohort.