Log in Sign up
Security Inside Meta's Law Enforcement Outreach Team, Where AI Flags the Threats and Humans Decide Whether to Call the Police News Silicon Photonics Supplier Xihe Technology Wins CSRC Approval to Raise 1.83 Billion Yuan on the STAR Market Business Ciena Says AI Demand Is Outrunning Supply as Its Backlog Tops 8.5 Billion Dollars and Heads Past 10 Billion Coding Assistants Swapping Coding Agent Harnesses Barely Moves Pass Rates but Can Triple the Bill, Shandong University Study Finds Microsoft Copilot Sweden's Government Offices Will Block ChatGPT and Claude on Work Computers From Monday, Pointing Staff to Copilot Instead Business Nvidia, Intel and AMD Have Prepaid 5 Billion Dollars to Taiwan's Substrate Makers as the AI Buildout Moves Upstream Security Anthropic Details How Mythos 5 Tried to Slip Malware Into a Real GitHub Project During a UK Security Test Business SK Hynix Halts Solidigm's US IPO and Will Fund a New American NAND Fab Itself, Saying the Market Would Shortchange the Listing Security Inside Meta's Law Enforcement Outreach Team, Where AI Flags the Threats and Humans Decide Whether to Call the Police News Silicon Photonics Supplier Xihe Technology Wins CSRC Approval to Raise 1.83 Billion Yuan on the STAR Market Business Ciena Says AI Demand Is Outrunning Supply as Its Backlog Tops 8.5 Billion Dollars and Heads Past 10 Billion Coding Assistants Swapping Coding Agent Harnesses Barely Moves Pass Rates but Can Triple the Bill, Shandong University Study Finds Microsoft Copilot Sweden's Government Offices Will Block ChatGPT and Claude on Work Computers From Monday, Pointing Staff to Copilot Instead Business Nvidia, Intel and AMD Have Prepaid 5 Billion Dollars to Taiwan's Substrate Makers as the AI Buildout Moves Upstream Security Anthropic Details How Mythos 5 Tried to Slip Malware Into a Real GitHub Project During a UK Security Test Business SK Hynix Halts Solidigm's US IPO and Will Fund a New American NAND Fab Itself, Saying the Market Would Shortchange the Listing

Swapping Coding Agent Harnesses Barely Moves Pass Rates but Can Triple the Bill, Shandong University Study Finds

Coding Assistants · By: AIReport Admin · 1 hour ago · 👁 9
Swapping Coding Agent Harnesses Barely Moves Pass Rates but Can Triple the Bill, Shandong University Study Finds

Researchers at Shandong University ran five models through three shipped agent harnesses and found that swapping harnesses moves SWE-bench results about as much as rerunning the same setup, while a single task can cost up to three times more. Claude Code sends roughly 16,600 tokens of system prompt and tool descriptions on every step, against about 7,000 for OpenCode and under 1,000 for mini-SWE-agent.

A coding agent is two things: a language model and a harness, the system prompt, tools, and context management that let the model work inside a code repository. New harnesses ship every week, and each vendor promises score gains over the last one. Two researchers at Shandong University, Liu and Han, decided to measure how much a harness actually buys when the model is held constant. Their paper has a title that gives the game away: "What does a harness buy? Mostly tokens." Their answer, from a controlled comparison posted as arXiv preprint 2610.04433, is that swapping harnesses moves results about as much as running the same setup twice, while the bill can triple.

The pair ran five models — Qwen3.6, Qwen3.8, DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview — through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent, a deliberately minimal harness that gives the model nothing but a Bash tool and skips context management entirely. Claude Opus 5 ran as a reference point inside Claude Code only. The test grid used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more, and several setups were run twice specifically to measure noise.

That noise turns out to be the story. Reruns of an identical setup already vary by up to six points on SWE-bench Verified, which means many published harness comparisons are measuring it. Earlier studies report gaps of more than 20 points between harnesses, but almost always from a single run. Zhipu AI advertises a 2.4-point lead for its own harness over Claude Code — a margin that falls well inside the noise band. Task-level flip rates tell the same story: between two reruns, the median outcome changes on 13 percent of tasks; between two different harnesses, it also changes on 13 percent. Swapping models pushes that to 22 percent.

On the 447-task pool, the heaviest and lightest harnesses, Claude Code and mini-SWE-agent, finished less than two points apart — statistically equivalent inside a five-point band. The only measurable harness effect was a steady deficit for OpenCode, which trailed the other two by up to nine points. About half of that gap traces to output-limit handling: when a model hits the limit mid-step, Claude Code prompts it to continue while OpenCode lets the step die, and the attempt usually fails. The authors flag the remainder as unexplained, noting OpenCode also imposed a much tighter time limit on locally run models.

Where the harness does show up is the invoice. The system prompt and tool descriptions are resent with every step, and they run about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. With the same model, tasks, and price list, a single task can cost up to three times more depending on the harness. On GLM-5.3-Flash, one trial cost 1.81 yuan in Claude Code against 0.56 yuan in OpenCode; cached-token discounts dampen the gap but do not change the ranking.

The study also stumbled onto a benchmarking hazard of its own. In early runs with web access, 71 percent of DeepSeek and Qwen attempts pulled the correct fix straight from GitHub instead of solving the task, while Claude models almost never did. The researchers cut off web access for all remaining runs rather than claim anyone trained on the test set. The one place harness choice genuinely changed results was reasoning: without extended thinking, mini-SWE-agent with Qwen3.6 lost eleven points while Claude Code and OpenCode held steady. A year-old Claude Code configuration solved a similar number of tasks at less than half the cost, because thinking was not on by default back then.

The conclusion the authors draw is blunt: a harness buys "fewer ways to lose a task, and a bill, not pass rate." With 45 tasks, even a 13-point gap is detectable only half the time; with 447, gaps near five points become reliable. They recommend publishing a rerun of the same setup alongside every harness comparison, and have released their data and analysis scripts on GitHub. A cross-check on Terminal-Bench 2.1 showed the same pattern. For an industry spending heavily on scaffold engineering — from Nvidia's Sol PI steering work to frameworks that promise double-digit gains — the paper is a reminder that the leaderboard deltas worth paying for may be smaller than the noise they arrive in.

Comments (0)

Log in to join the discussion

Log In

No comments yet