HD Moore, chief research officer at security firm Dragos and the creator of Metasploit, spent July through November 2026 sending nearly two million AI prompts, each containing a unique, never-before-used secret: AWS access keys tied to empty accounts he controlled, SSH passwords, unique URLs, DNS names and references to private git repositories. Each secret went into exactly one prompt sent to one provider, so any later use of it anywhere on the internet traced back to the exact prompt, model, router and privacy flag that carried it. He published a public summary of the results on November 19, and Ars Technica reported the findings on November 20.
The most serious finding: at least three AI inference providers had been compromised by attackers who used their access to siphon customer prompts. In one case, attackers reused "a couple hundred" of the seeded AWS keys — every request Moore sent during a few short windows in late October was stolen, including some requests the provider rejected before a model ever ran, which told Moore the copying happened at the provider's front door. Some stolen keys were tested within hours. Across the campaign, the bait keys generated tens of thousands of AWS API calls as intruders browsed S3 buckets, secrets managers and parameter stores, tried to enable hosted models in AWS Bedrock and added their own IAM users to maintain access. Secrets stolen from two of the breached providers were used together, almost immediately, from the same rented server — strong evidence the same adversary was behind both.
The second finding concerns the reseller layer. Many AI applications route prompts through brokers such as OpenRouter or Vercel, which read the plaintext to pick a model and may forward the request through further routers, resellers and hosting providers — each hop able to log the prompt. Moore found multiple resellers on inference marketplaces that lured customers with low prices, switched on request logging and passed prompts along to third parties. He identified them by sending identical requests to the same model through different sellers: when credentials were reused through one seller and not another, the leak source was obvious.
The third finding is that privacy flags are frequently not honored. A Chinese AI lab fetched the secret URLs embedded in more than half of the prompts sent to its models between July and October, including roughly two-thirds of those marked "no data collection" or Zero Data Retention; one seeded URL surfaced in the lab's internal company chat 24 days after the original request. Several US hosting providers that resell that lab's models triggered the same behavior, and prompts submitted by US users reached Chinese AI organizations. In a separate incident covered in the research reporting, Fireworks AI told Uber on October 9 that an unauthorized party had accessed Uber's account using compromised credentials, two days after the data was reached. Moore's conclusion is blunt: a Zero Data Retention promise is a policy claim, not a technical guarantee, and it is only as strong as the weakest party handling the data.
The detection gap worries Moore most. The three provider breaches went on for months, in one case back to mid-July, undetected until an outside researcher spoke up; providers learned of the compromises only when he notified them in November. As of late October, stolen keys were still in active use, one victim had published a preliminary notice, and another had not disclosed at all. All effectiveness figures in the study come from the researcher's own instrumentation, though the method — canary tokens at a scale of two million attributable prompts — turns years of anecdote into measurement.
Why it matters: anyone who pastes credentials, API keys, private code or internal links into an AI tool is trusting an entire supply chain they cannot see. Moore's recommendations are practical: self-host sensitive workloads where possible, go directly to a first-tier provider instead of a reseller marketplace, pin your router's upstream providers, enable secret filtering, and seed your own canaries so you find out what your providers actually do with your prompts. The era of trusting a privacy checkbox in an inference marketplace is over.
Comments (0)
Log in to join the discussion
Log InNo comments yet