5,625 Trials, 359 Out of 360: Coding Agents Copy the Codebase Instead of Following Your Instructions

A preprint posted to arXiv on 28 October asks a question that anyone who has handed a coding agent an established repository has run into: when two implementations both pass the tests, which one does the agent pick, and what makes it change its mind? The answer, from 5,625 trials across four agent and model combinations, is that the code already in the repository wins almost every time.

The authors — Haichao Zhu, Geyang Xu and three co-authors listing affiliations at UC Riverside and Tulane University — built a Flask task-manager API and chose six decisions where both options pass the same hidden tests: project_id against projectId, an integer id against a string id, a flat JSON response against one wrapped in a data object, a returned 404 against a raised NotFound. Because the tests cannot tell the options apart, the agent's choice reveals its preference rather than its correctness. The study ran 375 cells with 15 fresh sessions each, using Claude Sonnet 4.6 and Claude Opus 4.8 through Claude Code and two GPT models through the Codex CLI.

With nothing to copy, all four agents converged on the same defaults: integer identifiers, flat responses, direct error returns. Then the researchers planted the opposite convention in the existing code. The agents followed the repository in 359 of 360 conflicting trials. Intent, the third stage, was where the defaults reasserted themselves: when the task explicitly asked for something different from the existing code, the agents needed a written rule, not an example, to switch.

Placement turned out to matter more than volume. Adoption of a repository convention rose with the number of supporting files for the same resource: zero percent with none, 21 percent with one, 49 percent with two and 96 percent with three. The same three example files attached to a sibling resource moved adoption to 4 percent. In other words, an agent reading a codebase treats locality as evidence of intent, and it discounts examples that live somewhere else in the tree.

The intervention that worked was cheap. Showing the agent an example of the new convention succeeded in 38 of 60 runs; writing the requirement down as a rule that takes precedence succeeded in 59 of 60. Across four open-source repositories in Python, Go and TypeScript, adding the rule to the agent's instruction file raised success from 0 of 120 runs to 117 of 120.

The caveats are real and the authors state them. This is version one of a preprint and has not been peer reviewed. Much of the evidence comes from a single purpose-built API, which controls the experiment but is not the same as a large production codebase with years of accumulated inconsistency. The six decisions are stylistic rather than architectural, so the finding says more about convention drift than about whether an agent can restructure a system correctly.

For teams running agents against real repositories, the practical reading is that an instruction file is not documentation, it is configuration. A convention that exists only as a pattern in half the codebase will be copied inconsistently, because the agent infers intent from what is nearest to the file it is editing. If a rule matters, it has to be written where the agent reads it, and it has to say that it overrides the surrounding code.

The result also reframes a common complaint about agents producing code that "does not match our style". In most of these trials the agent was not ignoring the team's style; it was reproducing whichever style was closest, which is often legacy. That is a governance problem disguised as a model-quality problem, and it is fixable with a paragraph rather than a fine-tune.