Taste-Bench From Microsoft and CityU Shows AI Agents Pick the Right Long-Term Move Only 60 Percent of the Time

AI agents now run unsupervised for days, writing research code and building software projects, and along the way they make dozens of decisions whose true cost only surfaces much later: which hypothesis to test, which implementation to build on, which experiment to run next. A clean architecture and a sloppy one can pass the same tests today. Researchers from Microsoft and City University of Hong Kong call the ability to choose well an agent's taste, and in a paper released October 18 they built a benchmark to measure it.

Taste-Bench works on hindsight. The authors mined thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public MALT transcript release, for decision forks: points where an agent's path could have gone two ways and the later trajectory reveals which choice actually paid off. Each of the 502 questions freezes the run at the fork, hides everything after it, and asks a model to pick the better path. Two fork types are included, one from independent attempts at the same task that diverged, one from an agent that corrected itself mid-run, which tests whether a model can spot a mistake earlier than the agent that made it. Human review confirmed almost all auto-generated labels.

The results are sobering. Fourteen current models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax and Mistral families were tested, each question twice with options in both orders, and a question only counted if the model was consistent. Pure guessing scores 25 percent. The best model, GPT-5.6 Sol, averaged 59.7 percent, with GPT-5.5 at 59.5, and everything else well below. On a binary choice, the frontier barely beats a coin flip.

The timing of evidence explains most of the damage. When the clue that reveals the better path is already visible in the prior steps, models averaged 62.3 percent. When the evidence only appears after extensive later work, accuracy fell to 21 percent, near random. More reasoning budget did not help: across three reasoning settings on two models, accuracy barely moved, and models spent the most thinking tokens on exactly the questions they got wrong, which the authors read as the needed clue simply not existing yet at those forks.

The useful surprise is that taste can be taught. The team distilled the judgment of a teacher model that has seen the full trajectories into a Qwen3.6-27B student that only sees the question, lifting its accuracy from 30 to 47.9 percent on unseen tasks. Fed back into a separate solving agent as advice at each fork, that judgment raised the success rate on 41 held-out SWE-bench Pro tasks from 14.6 to 33.7 percent, capturing most of the 39 percent ceiling that perfectly correct advice would have achieved.

The benchmark lands in a broader credibility problem for end-to-end agent scores. METR has already shown that SWE-bench Verified significantly overestimates coding agent performance, with professionals rejecting many solutions that pass automated tests, and GPT-5.6 Sol itself posted the highest rate of cheating on software tests METR had ever recorded. Taste-Bench captures a different axis than those tests, and models that score similarly on SWE-bench Verified show clear gaps on it.

The benchmark and code are public, on Hugging Face as wenbopan/taste-bench and GitHub as wbopan/tastebench. The uncomfortable takeaway for the industry is that the skill separating an agent that burns its budget from one that ships is not measurable by whether the final tests pass, and currently sits near chance for almost everyone. If long-running autonomy is the product, taste, not raw capability, is the metric to watch, and right now it is trainable, which is the one genuinely optimistic number in the paper.