Evaluate coding agents on your repository with a reusable task card

Compare coding agents on the same repository tasks, starting revisions, acceptance criteria, and operating limits. Repeat the trials, retain failures, and inspect the resulting patches. The useful result is a bounded answer about your workflow—not a universal ranking of models.

A leaderboard can help choose candidates. It cannot tell you whether an agent will handle your migration conventions, slow integration suite, or ambiguous internal API. Anthropic’s evaluation guidance makes an important distinction: the evaluated agent includes both the model and the harness that supplies its tools and working loop. Record that combination, not merely the model name. ^[1]

Choose the decision before the tasks

An October Reddit discussion captures the confusion. Rebelfan26 describes setups taking different paths through model calls, repository reads, and tests. Replies distinguish isolating an API provider from comparing native agent workflows. The thread supplies a methodological problem, not a benchmark or ranking. ^[2] If provider latency is your question, hold the model and harness fixed. If tool adoption is your question, compare the complete configurations your team would use and name them accordingly.

Consider a hypothetical team deciding which agent should handle small backend bug fixes. The team wants patches that pass its normal review without interactive rescue.

Our proposed pilot starts with several representative closed issues: input validation, an incorrect query, an API compatibility bug, and a small state transition. Restore each repository to a revision before the fix. Remove access to the later solution, including reachable history or cached work that would reveal it. Use only code and data you are authorized to expose to the candidate service.

Do not choose the set by running agents first and keeping the interesting failures. Select tasks before comparing candidates and record exclusions. A task that no longer builds can be repaired or excluded under a consistent rule; quietly dropping it after one candidate fails changes the comparison.

Ten tasks with three trials each can be a practical exploratory pilot if the budget permits. Those numbers are a proposed starting design, not a statistical guarantee. Thirty attempts on ten tasks still represent ten distinct tasks.

Write a task card a reviewer can actually grade

For the hypothetical query bug, the requirement might be: archived projects must be excluded from the default listing, while an explicit archive filter continues to work. The acceptance criteria should say that. A hidden test should not introduce an undocumented requirement about a particular helper name.

This is a proposed card for local evaluation. A known working solution should pass the acceptance checks, and the original broken revision should fail the bug-specific check. That exercises the grader before it evaluates an agent.

Keep normal development tests visible. Hold back selected acceptance cases when you want to measure generalization beyond those examples. The behavioral requirements remain visible; only the evaluator’s exact test instances are withheld. Anthropic likewise warns that ambiguous tasks and rigid graders can reject valid work. ^[1]

Separate the patch from the judge

Have the agent produce a patch in one disposable environment. Apply that patch to a clean evaluator environment containing the trusted checks. Otherwise, changing a test, fixture, or test command can change the measurement alongside the proposed fix.

SWE-bench’s official harness applies generated patches and runs repository tests in Docker environments. Its result artifacts include the patch, test output, and harness log. That separation is a useful design reference for your own evaluator; it does not mean every private repository can be dropped into SWE-bench unchanged. ^[3]

Keep the evaluator isolated from production credentials and data. It still executes candidate code. A container makes the environment easier to reproduce; its existence alone does not establish a security boundary suitable for every hostile workload.

Check the final artifact rather than relying on the agent’s closing message. For the project-listing bug, inspect the query and run both default and explicit-archive cases. A transcript saying “all tests passed” is useful navigation, not the acceptance result.

Repeat the comparison without recycling state

Start every trial from the same clean task state. Clear writable caches and prior outputs that could expose a solution. Record each candidate’s complete settings and interleave candidate runs where practical, so a temporary outage or a different machine load does not affect only one side.

Multiple trials expose variability. They do not repair an unrepresentative task set. Report the number of successful attempts per task, then the aggregate; a tool that reliably fixes validation bugs but repeatedly fails migrations deserves a more specific conclusion than its average suggests.

If using SWE-bench’s harness, its documented result cache is keyed by run ID and instance ID. Reusing those identifiers can reuse an earlier result even when the submitted diff changes. Give genuinely new evaluations distinct run IDs. ^[3]

Choose the budget rule to match the purchasing decision. Equal wall-clock limits answer a different question from equal API-spend limits. Either can be useful, but switching rules after seeing the winner makes the result difficult to interpret.

Keep infrastructure failures in the record

A container that never starts is different from a patch that fails a valid assertion. Track both. Our proposed report has four primary outcomes: accepted patch, behavioral failure, review rejection, and infrastructure failure. Preserve the raw reason so ambiguous cases can be revisited.

Show acceptance across all scheduled attempts and, separately, across attempts that reached a valid evaluation. The first describes the operating experience; the second helps investigate coding capability. Do not erase infrastructure failures from the cost or elapsed-time account.

Count unsuccessful attempts and retries when calculating API cost per accepted patch. Record human review and recovery time separately unless you have explicitly chosen a monetary conversion. Our harness-cost analysis^[4] explains why a model price list leaves that task budget unresolved.

Eric Zakariasson describes a related Cursor workflow in a September X post: measure completed-task cost while holding quality constant. It is a vendor operational account, not an independent dataset; its reported savings are not targets for this pilot. ^[5]

Do not confuse agent success with developer productivity

METR’s February 2026 update illustrates why the distinction matters. Its follow-up developer study faced selection effects because participants avoided tasks they did not want to do without AI. Concurrent agents also complicated time reporting. The authors said the new data was an unreliable signal of the current productivity effect. ^[6]

A frozen-task agent trial avoids some of those problems but answers a different question. It does not measure how developers change task choice, learn a codebase, or use idle time while agents run. Do not translate its pass rate directly into a percentage productivity gain.

Before trying another tool, fill in one task card and prove that its grader distinguishes the broken revision from a valid fix. That reusable task is more valuable than an impressive demonstration you cannot run again after the next model or harness update.