When you wire Claude's own SDK or OpenAI's Codex harness into your agentic coding pipeline, you are betting that the vendor's wrapper knows its model best. A new contamination-controlled benchmark says that bet does not pay off. The coding agent harness effect, the gap between a vendor-native harness and a neutral multi-provider one on the same model, is statistically indistinguishable from zero. Claude Opus 4.8 solved 48.8% of 80 tasks under its native claude-agent-sdk and 50.0% under the neutral deepagents harness, according to the paper published on arXiv on September 8, 2026. The paired difference was -1.25 percentage points with a 95% confidence interval spanning -10.0 to +7.5. GPT-5.5 showed the mirror image: 55.6% under the openai-codex SDK versus 54.4% under neutral, a 1.25-point edge that is equally flat. The result challenges a default assumption in the agent-building community: that the vendor-native pairing is the safe, high-performance choice.
What exactly did the study measure?
An agentic coding system is two things stitched together: a language model and a harness. The harness is the tools, prompts, truncation policies, and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners commonly assume that tuning translates into more solved tasks. The author, Mohsen Arjmandi, designed the study to measure that assumption directly.
The task suite is private and contamination-controlled: 256 tasks total, comprising 179 repository tasks mined from four production codebases (AccessCtl, IdentityApp, FieldSvc, BookingSvc) with hidden tests and gold patches, plus 77 contest tasks published after a mechanically derived eligibility date. A machine-readable cutoff registry was frozen before data collection, and a runtime drift gate verified the served model identity before every scored phase. The tasks stay private to preserve the suite for future testing.
The 80 tasks in the paired contrast ran under two conditions each. On Claude Opus 4.8, the native harness was claude-agent-sdk and the neutral harness was deepagents, built on LangGraph. On GPT-5.5, the native harness was the openai-codex SDK and the neutral harness was the same deepagents setup. Gemini-3.5-flash and DeepSeek-v3.2 ran as side cells for context. Every run executed in its own KVM microVM with an append-only event ledger, and 792 of 800 planned runs were graded by a Docker-isolated oracle. The orchestrator, grading oracle, reanalysis code, and derived aggregates are all released; only the task content stays private.
This is not the first paper to question the harness assumption. A related arXiv study from August 2026 asked whether changing the harness changes the result when the model and task stay fixed, and reached a compatible conclusion. A KDD 2026 workshop paper found that harness choice induces up to a 40x difference in tokens per solved task across three open-source harnesses, reinforcing that the harness is a major hidden variable in agent evaluation.
Did either harness actually win?
No. Not on average. The paired differences are tiny and their confidence intervals cross zero on both models. Opus 4.8 came in at 48.8% native against 50.0% neutral, a gap of 1.25 percentage points. GPT-5.5 reversed the sign at 55.6% native against 54.4% neutral, also 1.25 points. Neither interval excludes zero.
What the averages hide is a sharp interaction effect on the Opus runs. The chart below shows the native-minus-neutral solve-rate difference split by task type.
On the 61 repository tasks, the native harness trails by 9.0 percentage points (95% CI -17.2 to -0.8). On the 19 contest tasks, it leads by 23.7 percentage points (CI +2.6 to +44.7). A label-permutation test gives p = 0.003 for this interaction. The author is explicit that this partition was chosen after seeing the data, so it is a post-hoc pattern that a designed replication should test, not a confirmed finding.
The read is still interesting even with that caveat. It suggests the vendor's harness tuning may optimize for the kind of self-contained, well-specified problems that benchmarks tend to use, meaning contest tasks, at the expense of messier real-world repository work. If your production workload looks more like fixing a bug in a monorepo than solving a competitive programming problem, the native harness might be working against you.
A separate finding reinforces the gap between benchmark correctness and autonomous completion. Of the 81 runs that were cancelled at the wall-clock ceiling, 22 had already produced a passing patch. The harness gave up on a solution that was already correct because it hit a time limit. That is a harness policy problem. Your agent's time budget settings could be quietly costing you solved tasks.
What does the harness choice cost you?
The cost story is where the harnesses separate, but not in the direction you might expect. The neutral harness was more expensive per solved task on both models. Repriced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8, with task-bootstrap intervals of 1.1 to 2.1 across pricing bases. On GPT-5.5 the neutral harness cost 1.2 times as much per solved task (CI 1.05 to 1.33).
| Metric | Opus 4.8 native | Opus 4.8 neutral | GPT-5.5 native | GPT-5.5 neutral |
|---|---|---|---|---|
| Solve rate | 48.8% | 50.0% | 55.6% | 54.4% |
| Paired diff | -1.25 pp | +1.25 pp | ||
| Cost per solved task | 1.0x | 1.3 to 1.6x | 1.0x | 1.2x |
The cost gap comes from behavior, not pricing. The neutral harness reached the wall-clock ceiling more often and made roughly twice the tool calls per run. More calls means more tokens, and more tokens means more spend per task, even when the solve rate is comparable.
There is an important caveat on the Opus cost figure. On the Anthropic account, 58 runs left no usage record in the telemetry. Allocating that unrecorded spend entirely to either cell would move the Opus cost ratio between 0.7 and 2.3, so the billed ordering on Anthropic is unresolved. The paper was revised in September 2026 specifically to correct a usage-semantics defect in the cost telemetry from the August 2026 manuscript, and the author is transparent that the cost results are observed-usage estimates, not settled numbers.
What should this change about your agent stack?
The practical takeaway depends on what you are optimizing for.
- If you are shipping on a single vendor's model, the native harness is the cheaper choice and there is no evidence it solves fewer tasks. Keep using it. The cost penalty of switching to a neutral harness, 1.2 to 1.6x per solved task, buys you nothing in capability on this data.
- If you are building a multi-model pipeline that routes across providers, the neutral harness is your only real option, and this data says you are not paying a capability tax for it. You are paying a cost tax, which you can manage with tighter tool-call budgets, better truncation policies, and shorter time ceilings.
- If your workload is heavy on real-world repository tasks rather than contest-style problems, the post-hoc stratum finding should make you skeptical of the native harness's tuning. The 9-point deficit on repository tasks is suggestive even if it needs replication. Test your own workload against both harnesses before committing.
- If you are benchmarking agents for a purchase decision, stop reporting single-number solve rates without confidence intervals. The 1.25-point differences here are inside noise, and a vendor leaderboard that reports them as rankings is misleading. This connects to a broader problem with LLM benchmark variance that the field has been underestimating.
The 22 passing-but-cancelled runs are the most actionable finding for any team running agents in production today. If your harness kills a run at a time limit without checking whether the last tool call already produced a valid result, you are wasting solved work. Audit your harness's completion policy. A cheap "check the latest patch before cancelling" step could recover a meaningful fraction of runs you are currently throwing away.
The cost data also connects to the economics of agent-heavy coding workflows where per-task spend is the binding constraint. If the neutral harness costs 1.6x per solved task, that multiplier can erase the savings from model routing or quantization. Model the full harness-inclusive cost, not just the per-token price.
The harness is the lever, not the model
The vendor-native harness is a cost optimization with a tuning bias. The solve rates differ by 1.25 percentage points in opposite directions across the two models; the neutral harness just spends 1.2 to 1.6 times as much to get there. The engineering decision that matters is how to tune the one you have: time budgets, tool-call limits, and completion checks are where you win or lose tasks. Those are your knobs to turn, and this paper suggests the vendor's tuning of them is no better than yours would be.
Sources
- arXiv - Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
- arXiv - Same Model, Different Harness: Different Coding-Agent Results
- KDD 2026 Workshop - The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
