Building a SWE-bench-Style Eval Harness for Your Own Coding Agent

Every few weeks another leaderboard update goes around: some frontier model just hit a new high score on SWE-bench Verified or Terminal-Bench. Teams read the headline number, pick the top model, and wire it into their internal coding agent — then find out three weeks later that it's making the same class of mistakes on their codebase that it supposedly "solved" on the benchmark.

The gap is real. SWE-bench tasks are drawn from a fixed slice of popular open-source Python repos (Django, scikit-learn, sympy, and a few dozen others). A model that scores 90% there has seen thousands of GitHub issues and pull requests from those exact repos during pretraining, and the tasks reward a specific shape of problem: a well-isolated bug with a clear reproduction case and an existing test suite. Your codebase is not Django. Your bugs are not isolated. Your test coverage is probably worse. None of that shows up in a public leaderboard number.

The fix isn't to distrust benchmarks — it's to build a small, private one scoped to how your team actually uses its coding agent. This post walks through building that harness: task format, an execution loop, scoring, and how to read the results without fooling yourself.

Why public benchmarks don't transfer

Three structural reasons a SWE-bench score doesn't predict your agent's performance on your repo:

1. Contamination risk. Popular benchmark repos are heavily represented in training data. A model can pattern-match a fix from memory rather than reason about the actual bug. Your internal repo, by definition, has never been seen.
2. Task shape mismatch. SWE-bench tasks are single-issue, single-PR, with a pre-existing regression test. Real engineering work is often multi-file, spans unrelated systems, and has no test until you write one — which is a different skill than the benchmark measures.
3. Environment mismatch. Your agent has to navigate your build system, your internal libraries, your lint rules, your CI quirks. None of that generalizes from a benchmark built on pip install and pytest.

This is the same argument covered in LLM Evals for Production Agents for agent evals generally — the principle here is identical, just applied specifically to coding agents: the only score that matters is the one measured on your own distribution of tasks.

Step 1 — Collect real tasks, not synthetic ones

Resist the urge to hand-write toy bugs. Mine your own history instead — it's the highest-signal, lowest-effort source:

• Closed GitHub/GitLab issues with a linked PR and a test that was added or modified in that PR
• Bug reports from your issue tracker with a reproduction script
• Refactor tickets with a clear "before" and "after" state

Aim for 30–50 tasks to start. That's enough to see meaningful pass-rate deltas between agent configurations without spending days curating.

Each task needs four things: the starting commit, the task description (the issue text, verbatim — don't clean it up, real prompts are messy), a golden patch (the actual merged PR diff), and a verification command (the test(s) that must pass after the fix).

# task_schema.py
from dataclasses import dataclass

@dataclass
class BenchTask:
    task_id: str          # e.g. "billing-svc-1423"
    repo_path: str        # local path or git URL
    base_commit: str      # commit SHA before the fix
    problem_statement: str  # the raw issue/ticket text
    golden_patch: str     # unified diff of the real fix, for reference/debugging
    test_cmd: str         # e.g. "pytest tests/billing/test_invoice.py -x"
    fail_to_pass: list[str]  # test names that must go from failing -> passing
    pass_to_pass: list[str]  # test names that must stay passing (no regressions)

Step 2 — Isolate execution

Each task run needs a clean, disposable environment: checkout at base_commit, apply nothing, hand the repo to the agent, then run tests. Never run this against a shared dev environment — parallel task runs will corrupt each other's state, and a bad agent patch could touch files outside the task scope.

# runner.py
import subprocess
import tempfile
import shutil
from pathlib import Path

def prepare_workspace(task: BenchTask) -> Path:
    workspace = Path(tempfile.mkdtemp(prefix=f"bench-{task.task_id}-"))
    subprocess.run(
        ["git", "clone", task.repo_path, str(workspace)],
        check=True, capture_output=True,
    )
    subprocess.run(
        ["git", "checkout", task.base_commit],
        cwd=workspace, check=True, capture_output=True,
    )
    return workspace

def run_task(task: BenchTask, agent_fn) -> dict:
    workspace = prepare_workspace(task)
    try:
        # agent_fn gets the workspace path and task prompt, and is expected
        # to leave its edits committed or staged in that working tree.
        agent_fn(workspace, task.problem_statement)
        result = subprocess.run(
            task.test_cmd.split(),
            cwd=workspace,
            capture_output=True,
            text=True,
            timeout=300,
        )
        return {
            "task_id": task.task_id,
            "passed": result.returncode == 0,
            "stdout": result.stdout[-4000:],  # tail only, logs get huge
            "stderr": result.stderr[-2000:],
        }
    finally:
        shutil.rmtree(workspace, ignore_errors=True)

Two details matter here that are easy to skip and expensive to skip:

• Timeouts on everything. An agent stuck in a retry loop or a runaway pip install will silently eat your whole benchmark run. 300 seconds per task is a reasonable default; tune per repo.
• Always clean up the workspace, even on failure — finally isn't optional. Left-over temp checkouts accumulate fast if you're running this in CI.

If your agent runs as a container or has broader tool access (shell, file write, network), run prepare_workspace inside a sandbox — a fresh Docker container or an ephemeral VM — not just a temp directory. A coding agent with shell access can rm -rf or exfiltrate outside its intended scope if the task or a tool response contains adversarial content. See Sandboxing LLM-Generated Code Execution for the isolation patterns.

Step 3 — Score fail_to_pass and pass_to_pass separately

Don't just check "did the test suite pass." SWE-bench's actual scoring insight — worth copying directly — is separating two questions:

• FAIL_TO_PASS: did the specific test(s) that reproduce the bug go from failing to passing?
• PASS_TO_PASS: did all the tests that were already passing stay passing?

An agent that "fixes" a bug by deleting the failing test, or by making an unrelated change that happens to satisfy a loose assertion, will pass a naive test suite exit code == 0 check. It will not pass FAIL_TO_PASS/PASS_TO_PASS scoring, because you're checking specific named tests before and after.

def score_task(task: BenchTask, result: dict, baseline_test_results: dict) -> dict:
    # baseline_test_results: dict of {test_name: passed_bool} run once,
    # before any agent touches the repo, at base_commit.
    post_results = parse_test_output(result["stdout"])  # -> {test_name: bool}

    fail_to_pass_ok = all(
        post_results.get(t, False) for t in task.fail_to_pass
    )
    pass_to_pass_ok = all(
        post_results.get(t, False) == baseline_test_results.get(t, False)
        for t in task.pass_to_pass
        if baseline_test_results.get(t, False)  # only ones that passed at baseline
    )

    return {
        "task_id": task.task_id,
        "resolved": fail_to_pass_ok and pass_to_pass_ok,
        "fail_to_pass_ok": fail_to_pass_ok,
        "pass_to_pass_ok": pass_to_pass_ok,
    }

Run the test suite once at base_commit with no agent involved to capture baseline_test_results — you need that snapshot to detect regressions the agent introduces elsewhere in the codebase.

Step 4 — Aggregate and compare configurations

The point of the harness isn't a single number — it's comparing configurations: model A vs model B, one system prompt vs another, with vs without a particular tool. Run every configuration against the identical task set and diff the resolved sets, not just the pass rates.

def compare_runs(run_a: dict[str, bool], run_b: dict[str, bool], label_a: str, label_b: str):
    only_a = [t for t, ok in run_a.items() if ok and not run_b.get(t)]
    only_b = [t for t, ok in run_b.items() if ok and not run_a.get(t)]
    both = [t for t, ok in run_a.items() if ok and run_b.get(t)]

    print(f"{label_a} rate: {sum(run_a.values())}/{len(run_a)}")
    print(f"{label_b} rate: {sum(run_b.values())}/{len(run_b)}")
    print(f"Solved by both: {len(both)}")
    print(f"Only {label_a}: {only_a}")   # regressions if label_b is your new config
    print(f"Only {label_b}: {only_b}")   # gains if label_b is your new config

Two configs with identical pass rates can be solving entirely different tasks — a raw percentage hides that. The only_a / only_b lists are what tell you whether a new model or prompt is a strict improvement or a lateral trade.

Comparison: public benchmarks vs. your own harness

	SWE-bench / Terminal-Bench	Private harness on your repo
Task source	Curated open-source repos	Your closed issues + real PRs
Contamination risk	High for popular repos	None — model has never seen it
Reflects your build/lint/CI quirks	No	Yes
Comparable across the industry	Yes	No — it's yours
Setup cost	Zero (use the leaderboard)	A few hours to a day
Ongoing value	Tracks frontier model progress	Tracks your agent's fitness for your work

Use both. Public benchmarks are a reasonable first filter for picking which models to even try. Your private harness is what tells you whether the agent you've picked is actually ready to touch your codebase unsupervised.

Common pitfalls

• Cherry-picked tasks. If you hand-select "impressive" bugs, you'll overestimate the agent. Sample tasks the same way real work arrives — including boring ones.
• No baseline run. Skipping the pre-agent test snapshot means you can't detect regressions in pass_to_pass tests, which is often where agents quietly do damage.
• Testing only "resolved," ignoring partial credit. A patch that fixes the bug but breaks style/lint conventions your team cares about is a real failure mode — add a lint/format check as its own scored dimension, not folded into pass/fail.
• Running once and trusting it. LLM agents are non-deterministic. Run each task 3–5 times and report a pass rate, not a single boolean, especially for tasks near your decision threshold.
• Letting the harness rot. Tasks age — repos evolve past the base commit, dependencies drift. Refresh the task set quarterly or it stops representing current work.
• No isolation. Running agent patches directly against a shared checkout or without sandboxing is how a bad rm -rf or dependency install trashes a shared environment. Isolate every run.

Next steps

Start with 10 tasks pulled from your last month of closed bug tickets — that's enough to catch an agent config that's clearly wrong before you find out the expensive way, in production. Expand to 30–50 once the harness is stable, and re-run it every time you change the underlying model, system prompt, or tool set for your coding agent.

If you're building or evaluating AI coding agents as part of your day-to-day engineering work, Joinloop's AI Engineering cohort covers this kind of eval infrastructure in depth, alongside RAG, tool-calling, and production agent architecture.