Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Business Daniel Ek Is Backing Europa Fusion, a Stealth UK Startup Led by a Former DeepMind Researcher Betting on Pulsed Power Coding Assistants Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement Security A Transcription Firm Used by Police and Prosecutors Left Interview Transcripts and Body-Cam Footage on an Open S3 Bucket News UN Killer-Robot Talks End With a 2028 Deadline and No Binding Rules. About 100 States Wanted a Treaty; the US, Russia and Israel Blocked It Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Business Daniel Ek Is Backing Europa Fusion, a Stealth UK Startup Led by a Former DeepMind Researcher Betting on Pulsed Power Coding Assistants Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement Security A Transcription Firm Used by Police and Prosecutors Left Interview Transcripts and Body-Cam Footage on an Open S3 Bucket News UN Killer-Robot Talks End With a 2028 Deadline and No Binding Rules. About 100 States Wanted a Treaty; the US, Russia and Israel Blocked It

Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement

Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement

Stripe published a benchmark of 69 payments, subscriptions and Connect scenarios run against 14 model-and-harness pairs, twice each, with and without its own agent plugin. Agents averaged 78.1% of grading criteria, but only 29.4% of runs passed every mandatory requirement; the best pairing reached 52.9% strict. The plugin raised the pooled score to 82.6% and cut web searches per evaluation by 96%, while outdated API parameters fell from 7% to 1.3%.

Stripe has published a benchmark that measures something general coding tests cannot see: whether an AI agent that writes a working Stripe integration also writes a correct one. The company ran 69 public cases, split into 24 payments, 26 subscriptions and 19 Connect scenarios, against 14 combinations of model and coding harness, each case twice, with and without Stripe's own agent plugin. The headline is a split verdict. Agents averaged 78.1% of the grading criteria, but when every mandatory requirement in a case had to pass, the average fell to 29.4%.

The gap between those two numbers is the point. Each scenario was graded criterion by criterion, with MUST requirements separated from SHOULD recommendations, and judged against both the session transcript and the resulting workspace. Partial credit rewards an integration that mostly works. Strict pass punishes the last mile, and the last mile is where money moves: in one fulfillment case, an agent checked that a payment had succeeded but never checked whether the order was still eligible to ship after a refund or dispute.

Stripe's own plugin helped every pairing, but modestly. The plugin bundles Stripe-authored skills with an MCP server that offers documentation search and an implementation planner; across all 14 pairs it added roughly 3 to 8 percentage points, lifting the pooled score from 78.1% to 82.6%. Individual results moved unevenly: GPT-6.1 Sol inside Codex went from 85.6% to 89.6%, while Gemini 3.8 Flash inside OpenCode rose from 64.8% to 72.7%. The best configuration overall, GPT-6 Astra in Codex, averaged 90.0% and still passed every required criterion in only 52.9% of runs.

The most striking effect was on how agents searched. With the plugin in the tool list, web searches per evaluation fell 96%, from 1.26 to 0.05, and page fetches dropped 63%. Outdated parameter references fell from 7% to 1.3% of API call parameters. Stripe's explanation for the baseline problem is training data: agents default to the Stripe APIs that dominate their training corpora, so a model reached for a card-only parameter that had recently been removed until the plugin pointed it at automatic payment methods instead.

Behavior diverged by model rather than converging. Fable 5.1 made 53% more tool calls with the plugin available, while Gemini 3.8 Flash made 26% fewer, and agents that led on one-time payments did not necessarily lead on Connect, whose scenarios require more integration decisions and gained the most from the plugin. Runs were headless, in isolated workspaces, with a 45-minute timeout and timed-out errors excluded from scoring.

Three caveats sit inside the methodology. The benchmark grades specification and artifacts, not execution, so a well-shaped integration that has never run can still score full marks. A role-playing agent stood in for the developer and the criterion judge was GPT-6.1 Sol, one of the models under test, though Stripe says a probe using Opus as judge moved scores in lockstep and showed no bias, only a more lenient scorer. And the plugin being measured is Stripe's own product, evaluated on a benchmark Stripe designed, which makes the lift a vendor measurement rather than an independent one.

Stripe is publishing the test cases and its Forge evaluation harness on GitHub so others can rerun the suite and add models. The immediate value is less a leaderboard than an acceptance-test template: refunds, disputes, duplicate webhooks, subscription changes and reconciliation with order records are the checks that separate a demo checkout from one that can take money.

The broader signal for teams wiring agents into payments infrastructure is that a high partial-credit score is a ceiling, not a forecast. If the best configuration in a purpose-built, vendor-designed benchmark meets every requirement barely half the time, then agent-written billing code still needs the failure paths exercised by hand before it goes anywhere near a customer.

Comments (0)

Log in to join the discussion

Log In

No comments yet