Stripe has published a benchmark that measures something general coding tests cannot see: whether an AI agent that writes a working Stripe integration also writes a correct one. The company ran 69 public cases, split into 24 payments, 26 subscriptions and 19 Connect scenarios, against 14 combinations of model and coding harness, each case twice, with and without Stripe's own agent plugin. The headline is a split verdict. Agents averaged 78.1% of the grading criteria, but when every mandatory requirement in a case had to pass, the average fell to 29.4%.
The gap between those two numbers is the point. Each scenario was graded criterion by criterion, with MUST requirements separated from SHOULD recommendations, and judged against both the session transcript and the resulting workspace. Partial credit rewards an integration that mostly works. Strict pass punishes the last mile, and the last mile is where money moves: in one fulfillment case, an agent checked that a payment had succeeded but never checked whether the order was still eligible to ship after a refund or dispute.
Stripe's own plugin helped every pairing, but modestly. The plugin bundles Stripe-authored skills with an MCP server that offers documentation search and an implementation planner; across all 14 pairs it added roughly 3 to 8 percentage points, lifting the pooled score from 78.1% to 82.6%. Individual results moved unevenly: GPT-6.1 Sol inside Codex went from 85.6% to 89.6%, while Gemini 3.8 Flash inside OpenCode rose from 64.8% to 72.7%. The best configuration overall, GPT-6 Astra in Codex, averaged 90.0% and still passed every required criterion in only 52.9% of runs.
The most striking effect was on how agents searched. With the plugin in the tool list, web searches per evaluation fell 96%, from 1.26 to 0.05, and page fetches dropped 63%. Outdated parameter references fell from 7% to 1.3% of API call parameters. Stripe's explanation for the baseline problem is training data: agents default to the Stripe APIs that dominate their training corpora, so a model reached for a card-only parameter that had recently been removed until the plugin pointed it at automatic payment methods instead.
Behavior diverged by model rather than converging. Fable 5.1 made 53% more tool calls with the plugin available, while Gemini 3.8 Flash made 26% fewer, and agents that led on one-time payments did not necessarily lead on Connect, whose scenarios require more integration decisions and gained the most from the plugin. Runs were headless, in isolated workspaces, with a 45-minute timeout and timed-out errors excluded from scoring.
Three caveats sit inside the methodology. The benchmark grades specification and artifacts, not execution, so a well-shaped integration that has never run can still score full marks. A role-playing agent stood in for the developer and the criterion judge was GPT-6.1 Sol, one of the models under test, though Stripe says a probe using Opus as judge moved scores in lockstep and showed no bias, only a more lenient scorer. And the plugin being measured is Stripe's own product, evaluated on a benchmark Stripe designed, which makes the lift a vendor measurement rather than an independent one.
Stripe is publishing the test cases and its Forge evaluation harness on GitHub so others can rerun the suite and add models. The immediate value is less a leaderboard than an acceptance-test template: refunds, disputes, duplicate webhooks, subscription changes and reconciliation with order records are the checks that separate a demo checkout from one that can take money.
The broader signal for teams wiring agents into payments infrastructure is that a high partial-credit score is a ceiling, not a forecast. If the best configuration in a purpose-built, vendor-designed benchmark meets every requirement barely half the time, then agent-written billing code still needs the failure paths exercised by hand before it goes anywhere near a customer.
Comments (0)
Log in to join the discussion
Log InNo comments yet