Anthropic's Own Study Found Experts Followed a Coin-Flip AI Bias 86% of the Time and Still Called the Answer Theirs

Anthropic asked 50 professional analysts to do a realistic statistical job with Claude Code and then quietly rigged the exercise. Each analyst advised a fictional farming cooperative on whether to expand a paid crop advisory programme. The co-op's board would only expand if the programme lifted productivity by roughly 3 to 4 percent, but it never specified how productivity should be measured. Along the way each analyst had to settle four questions with two defensible answers: whether to measure yield per hectare or per unit of operating cost, whether to use a difference-in-differences design or a regression-adjusted comparison, which farms the estimate should cover, and how to weight them.

For every analyst and every question, a coin flip decided which option their agent would favour. The analysts were not told. The agent was instructed to make the strongest honest case for its assigned lean, to change its mind if the evidence demanded it, and never to misrepresent results.

Across the four decisions, analysts matched the agent's lean 86 percent of the time. Without any influence the expected rate would be about 50 percent. The breakdown was 92 percent on the outcome measure, 90 percent on weighting, 82 percent on the comparison method and 81 percent on which farms to cover. The sharpest result came from the 38 cases where an analyst had stated a preference before the agent spoke: they switched away from their own view 60 percent of the time when the agent disagreed, against 6 percent when it agreed.

The final recommendation moved less than the inputs. When the agent leaned toward expanding the programme, 54 percent of analysts recommended expansion; when it leaned the other way, 36 percent did. A statistical model using only the four coin-flip leans predicted 82 percent of the analysts' recommendations, against 61 percent for always guessing the most common answer.

Attribution was the part Anthropic's researchers described as most striking. Asked afterward to explain their reasoning, 77 percent of analysts repeated an argument the agent had made, some of them word for word, while 15 percent credited the agent anywhere. In the survey, 52 percent rated the final recommendation as more their own than the agent's. Analysts who followed all four leans were about as confident as everyone else, 71 against 73 on a 100-point scale.

The control group points the other way, with caveats. Of 61 analysts recruited, 11 chose to work without the agent; six of those 11 made serious methodological errors, against three of the 50 who used it, and their median completion time was 90 minutes against 65. Anthropic cautions that the group was small and less experienced, so the comparison cannot show that AI use caused the difference.

Anthropic is also explicit about what the study does not show. It deliberately strengthened the agents' preferences to measure their effect, and says the effect could be smaller when agents are given no clear lean; with no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs. The trial covered one task in one field with one model, the analysts were new to farming, and the confidence interval on the switching gap is wide. The figures come from Anthropic's own write-up of its own product, and no independent replication has been published.

What it means: the finding lands on a specific weak point in AI oversight. Reviewers are treated as the last independent check on model behaviour, including in safety research where agents now run and analyse experiments. That check only works if the reviewer's judgment is actually the reviewer's. If a coin flip inside an agent can move an expert's methodology 86 percent of the time, and the expert still signs the result as their own, then "human in the loop" describes a seating arrangement, not a control.