Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Business Daniel Ek Is Backing Europa Fusion, a Stealth UK Startup Led by a Former DeepMind Researcher Betting on Pulsed Power Coding Assistants Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement Security A Transcription Firm Used by Police and Prosecutors Left Interview Transcripts and Body-Cam Footage on an Open S3 Bucket News UN Killer-Robot Talks End With a 2028 Deadline and No Binding Rules. About 100 States Wanted a Treaty; the US, Russia and Israel Blocked It Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Business Daniel Ek Is Backing Europa Fusion, a Stealth UK Startup Led by a Former DeepMind Researcher Betting on Pulsed Power Coding Assistants Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement Security A Transcription Firm Used by Police and Prosecutors Left Interview Transcripts and Body-Cam Footage on an Open S3 Bucket News UN Killer-Robot Talks End With a 2028 Deadline and No Binding Rules. About 100 States Wanted a Treaty; the US, Russia and Israel Blocked It

2 Million Canary Secrets Sent Through AI Providers Found 3 Breaches, Ignored Privacy Flags and Keys Used Within Hours

2 Million Canary Secrets Sent Through AI Providers Found 3 Breaches, Ignored Privacy Flags and Keys Used Within Hours

Security researcher HD Moore seeded nearly 2 million prompts with unique, never-before-used secrets and watched where they surfaced: at least three AI inference providers were compromised by attackers siphoning prompts, marketplace resellers logged and passed on credentials, and one model lab fetched secret URLs from more than half of the prompts sent to it, including roughly two-thirds marked no-data-collection.

HD Moore, chief research officer at security firm Dragos and the creator of Metasploit, spent July through November 2026 sending nearly two million AI prompts, each containing a unique, never-before-used secret: AWS access keys tied to empty accounts he controlled, SSH passwords, unique URLs, DNS names and references to private git repositories. Each secret went into exactly one prompt sent to one provider, so any later use of it anywhere on the internet traced back to the exact prompt, model, router and privacy flag that carried it. He published a public summary of the results on November 19, and Ars Technica reported the findings on November 20.

The most serious finding: at least three AI inference providers had been compromised by attackers who used their access to siphon customer prompts. In one case, attackers reused "a couple hundred" of the seeded AWS keys — every request Moore sent during a few short windows in late October was stolen, including some requests the provider rejected before a model ever ran, which told Moore the copying happened at the provider's front door. Some stolen keys were tested within hours. Across the campaign, the bait keys generated tens of thousands of AWS API calls as intruders browsed S3 buckets, secrets managers and parameter stores, tried to enable hosted models in AWS Bedrock and added their own IAM users to maintain access. Secrets stolen from two of the breached providers were used together, almost immediately, from the same rented server — strong evidence the same adversary was behind both.

The second finding concerns the reseller layer. Many AI applications route prompts through brokers such as OpenRouter or Vercel, which read the plaintext to pick a model and may forward the request through further routers, resellers and hosting providers — each hop able to log the prompt. Moore found multiple resellers on inference marketplaces that lured customers with low prices, switched on request logging and passed prompts along to third parties. He identified them by sending identical requests to the same model through different sellers: when credentials were reused through one seller and not another, the leak source was obvious.

The third finding is that privacy flags are frequently not honored. A Chinese AI lab fetched the secret URLs embedded in more than half of the prompts sent to its models between July and October, including roughly two-thirds of those marked "no data collection" or Zero Data Retention; one seeded URL surfaced in the lab's internal company chat 24 days after the original request. Several US hosting providers that resell that lab's models triggered the same behavior, and prompts submitted by US users reached Chinese AI organizations. In a separate incident covered in the research reporting, Fireworks AI told Uber on October 9 that an unauthorized party had accessed Uber's account using compromised credentials, two days after the data was reached. Moore's conclusion is blunt: a Zero Data Retention promise is a policy claim, not a technical guarantee, and it is only as strong as the weakest party handling the data.

The detection gap worries Moore most. The three provider breaches went on for months, in one case back to mid-July, undetected until an outside researcher spoke up; providers learned of the compromises only when he notified them in November. As of late October, stolen keys were still in active use, one victim had published a preliminary notice, and another had not disclosed at all. All effectiveness figures in the study come from the researcher's own instrumentation, though the method — canary tokens at a scale of two million attributable prompts — turns years of anecdote into measurement.

Why it matters: anyone who pastes credentials, API keys, private code or internal links into an AI tool is trusting an entire supply chain they cannot see. Moore's recommendations are practical: self-host sensitive workloads where possible, go directly to a first-tier provider instead of a reseller marketplace, pin your router's upstream providers, enable secret filtering, and seed your own canaries so you find out what your providers actually do with your prompts. The era of trusting a privacy checkbox in an inference marketplace is over.

Comments (0)

Log in to join the discussion

Log In

No comments yet