When you wire Claude's own SDK or OpenAI's Codex wrapper into your agentic coding pipeline, you are betting that the vendor's wrapper knows its model best. A new contamination-controlled benchmark says that bet does not pay off. The coding agent wrapper effect, the gap between a vendor-native tool-and-prompt layer and a neutral multi-provider one on the same model, is statistically indistinguishable from zero. Claude Opus 4.8 solved 48.8% of 80 tasks under its native claude-agent-sdk and 50.0% under the neutral deepagents wrapper, according to the paper published on arXiv on September 8, 2026. The paired difference was -1.25 percentage points with a 95% confidence interval spanning -10.0 to +7.5. GPT-5.5 showed the mirror image: 55.6% under the openai-codex SDK versus 54.4% under neutral, a 1.25-point edge that is equally flat. The result challenges a default assumption in the agent-building community: that the vendor-native pairing is the safe, high-performance choice.
A model plus a wrapper
An agentic coding system is two things stitched together: a language model and a tool-and-prompt wrapper. The wrapper is the tools, prompts, truncation policies, and control flow that turn a chat model into an autonomous software engineer. Vendors ship these wrappers tuned to their own models, and practitioners commonly assume that tuning translates into more solved tasks. The author, Mohsen Arjmandi, designed the study to measure that assumption directly.
The task suite is private and contamination-controlled: 256 tasks total, comprising 179 repository tasks mined from four production codebases (AccessCtl, IdentityApp, FieldSvc, BookingSvc) with hidden tests and gold patches, plus 77 contest tasks published after a mechanically derived eligibility date. A machine-readable cutoff registry was frozen before data collection, and a runtime drift gate verified the served model identity before every scored phase. The tasks stay private to preserve the suite for future testing.
The 80 tasks in the paired contrast ran under two conditions each. On Claude Opus 4.8, the native wrapper was claude-agent-sdk and the neutral wrapper was deepagents, built on LangGraph. On GPT-5.5, the native wrapper was the openai-codex SDK and the neutral wrapper was the same deepagents setup. Gemini-3.5-flash and DeepSeek-v3.2 ran as side cells for context. Every run executed in its own KVM microVM with an append-only event ledger, and 792 of 800 planned runs were graded by a Docker-isolated oracle. The orchestrator, grading oracle, reanalysis code, and derived aggregates are all released; only the task content stays private.
This is not the first paper to question the wrapper assumption. A related arXiv study from August 2026 asked whether changing the tool-and-prompt layer changes the result when the model and task stay fixed, and reached a compatible conclusion. A KDD 2026 workshop paper found that wrapper choice induces up to a 40x difference in tokens per solved task across three open-source wrappers, reinforcing that the tool-and-prompt layer is a major hidden variable in agent evaluation.
Neither wrapper won on average
No. Not on average. The paired differences are tiny and their confidence intervals cross zero on both models. Opus 4.8 came in at 48.8% native against 50.0% neutral, a gap of 1.25 percentage points. GPT-5.5 reversed the sign at 55.6% native against 54.4% neutral, also 1.25 points. Neither interval excludes zero.
What the averages hide is a sharp interaction effect on the Opus runs. The chart below shows the native-minus-neutral solve-rate difference split by task type.
On the 61 repository tasks, the native wrapper trails by 9.0 percentage points (95% CI -17.2 to -0.8). On the 19 contest tasks, it leads by 23.7 percentage points (CI +2.6 to +44.7). A label-permutation test gives p = 0.003 for this interaction. The author is explicit that this partition was chosen after seeing the data, so it is a post-hoc pattern that a designed replication should test, not a confirmed finding.
The read is still interesting even with that caveat. It suggests the vendor's wrapper tuning may optimize for the kind of self-contained, well-specified problems that benchmarks tend to use, meaning contest tasks, at the expense of messier real-world repository work. If your production workload looks more like fixing a bug in a monorepo than solving a competitive programming problem, the native wrapper might be working against you.
A separate finding reinforces the gap between benchmark correctness and autonomous completion. Of the 81 runs that were cancelled at the wall-clock ceiling, 22 had already produced a passing patch. The tool-and-prompt layer gave up on a solution that was already correct because it hit a time limit. That is a wrapper policy problem. Your agent's time budget settings could be quietly costing you solved tasks.
Wrapper costs separate in a surprising direction
The cost story is where the wrappers separate, but not in the direction you might expect. The neutral wrapper was more expensive per solved task on both models. Repriced from raw per-turn usage at frozen list prices, the neutral wrapper cost 1.3 to 1.6 times as much per solved task on Opus 4.8, with task-bootstrap intervals of 1.1 to 2.1 across pricing bases. On GPT-5.5 the neutral wrapper cost 1.2 times as much per solved task (CI 1.05 to 1.33).
| Metric | Opus 4.8 native | Opus 4.8 neutral | GPT-5.5 native | GPT-5.5 neutral |
|---|---|---|---|---|
| Solve rate | 48.8% | 50.0% | 55.6% | 54.4% |
| Paired diff | -1.25 pp | +1.25 pp | ||
| Cost per solved task | 1.0x | 1.3 to 1.6x | 1.0x | 1.2x |
The cost gap comes from behavior, not pricing. The neutral wrapper reached the wall-clock ceiling more often and made roughly twice the tool calls per run. More calls means more tokens, and more tokens means more spend per task, even when the solve rate is comparable.
There is an important caveat on the Opus cost figure. On the Anthropic account, 58 runs left no usage record in the telemetry. Allocating that unrecorded spend entirely to either cell would move the Opus cost ratio between 0.7 and 2.3, so the billed ordering on Anthropic is unresolved. The paper was revised in September 2026 specifically to correct a usage-semantics defect in the cost telemetry from the August 2026 manuscript, and the author is transparent that the cost results are observed-usage estimates, not settled numbers.
It depends on what you optimize for
The practical takeaway depends on what you are optimizing for.
- If you are shipping on a single vendor's model, the native wrapper is the cheaper choice and there is no evidence it solves fewer tasks. Keep using it. The cost penalty of switching to a neutral wrapper, 1.2 to 1.6x per solved task, buys you nothing in capability on this data.
- If you are building a multi-model pipeline that routes across providers, the neutral wrapper is your only real option. You are paying a cost tax, which you can manage with tighter tool-call budgets, better truncation policies, and shorter time ceilings.
- If your workload is heavy on real-world repository tasks rather than contest-style problems, the post-hoc stratum finding should make you skeptical of the native wrapper's tuning. The 9-point deficit on repository tasks is suggestive even if it needs replication. Test your own workload against both wrappers before committing.
- If you are benchmarking agents for a purchase decision, stop reporting single-number solve rates without confidence intervals. The 1.25-point differences here are inside noise, and a vendor leaderboard that reports them as rankings is misleading. This connects to a broader problem with LLM benchmark variance that the field has been underestimating.
The 22 passing-but-cancelled runs are the most actionable finding for any team running agents in production today. If your wrapper kills a run at a time limit without checking whether the last tool call already produced a valid result, you are wasting solved work. Audit your wrapper's completion policy. A cheap "check the latest patch before cancelling" step could recover a meaningful fraction of runs you are currently throwing away.
The cost data also connects to the economics of agent-heavy coding workflows where per-task spend is the binding constraint. If the neutral wrapper costs 1.6x per solved task, that multiplier can erase the savings from model routing or quantization. Model the full wrapper-inclusive cost, not just the per-token price.
The wrapper is the lever, not the model
The vendor-native wrapper is a cost optimization with a tuning bias. The solve rates differ by 1.25 percentage points in opposite directions across the two models; the neutral wrapper just spends 1.2 to 1.6 times as much to get there. The engineering decision that matters is how to tune the one you have: time budgets, tool-call limits, and completion checks are where you win or lose tasks. Those are your knobs to turn, and this paper suggests the vendor's tuning of them is no better than yours would be.
Sources
- arXiv - Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
- arXiv - Same Model, Different Harness: Different Coding-Agent Results
- KDD 2026 Workshop - The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
