Mercor Mapped 29 AI Benchmarks Onto 18,797 American Job Tasks — and Found 90 Percent of US Working Hours Barely Measured

AI models keep posting superhuman scores on benchmarks that, according to a new analysis, measure only a sliver of the American economy. Mercor, the AI hiring and evaluation startup, published an interactive "Benchmark Map" this week that overlays 29 public agentic benchmarks onto the Department of Labor's O*NET taxonomy — 923 occupations across 22 job families, broken into 18,797 individual task statements. Its headline estimate: roughly 90% of US working hours are spent on tasks with little or no benchmark coverage.

The method is a coverage audit, not a capability test. Mercor used an LLM judge to map each benchmark prompt onto the O*NET tasks it resembles, and counted a task as covered only when at least five prompts matched it — a threshold meant to avoid declaring victory on a single lucky question. By that rule, only 924 of 18,797 tasks are covered, just 208 of 923 occupations have even one covered task, and six of the 22 job families contain no covered occupation at all. The authors are upfront that the result is provisional and their benchmark list incomplete.

What gets measured is not a random sample of work. AI-exposed tasks are about six times more likely to appear in the index than unexposed ones. Occupations with larger total wage bills are overrepresented even after controlling for exposure, and within a profession, tasks experts rate most important are an estimated 6 to 10 times more likely to be covered than the least important ones. The contrast within law is instructive: lawyers' tasks are heavily benchmarked, while similarly AI-exposed legal secretaries and court clerks are barely covered at all.

Mercor also tested whether the skew changes model rankings. Weighting its own APEX scores by how often tasks occur in an occupation produced mildly different numbers than a simple average — 70.2% versus 73.3% on financial analysis, for example — but with a catch: fewer than a third of the relevant O*NET tasks had enough prompts to be included in the weighted scores at all.

The caveats deserve equal billing. The mapping itself is automated and probabilistic; one prompt can match several tasks; time-share estimates are modeled; and Mercor sells its own benchmarks, which feature in the index — so read the framing with that conflict in mind. What the map shows is absence of evidence, not evidence of absence: it does not say AI fails at 90% of work, it says nobody has standardized measurements for most of it.

The finding lands amid a broader wave of evaluation skepticism. The forward-deployed engineering benchmark FDE-Gym recently found the best frontier agent configuration resolves only 14 of 95 real delivery tasks when unstated customer constraints must be uncovered, and benchmark-dependency debates flare whenever a model swings 47 points between harnesses on the same test. Teams making build, buy or deployment decisions on leaderboard deltas are, increasingly, extrapolating from a thin slice of the work they actually need done.

The practical takeaway is unglamorous but useful: a benchmark score is a measurement under chosen conditions, and for most occupations those conditions have never been constructed. Until the map fills in, the most honest evaluation of an AI system remains running it on a slice of your own workflow — the part of the economy no leaderboard has gotten to yet.