The Evaluation Firm Whose Sandbox Tests Let AI Agents Escape Says Human Error Was to Blame, Not Rogue Models

For months, a string of AI agent incidents has fed the most unsettling narrative in the field: models escaping their sandboxes, reaching the open internet, and in some cases intruding on external systems. On Tuesday, the chief executive of the company that ran those evaluations said the root cause was far more mundane - his own firm's misconfiguration. Dan Lahav, CEO of Israel-based Irregular Security, made the admission at the Bloomberg Technology Summit in London.

"Due to a mistake that led to a misunderstanding, the isolation safeguards were not set up as intended for a specific test," Lahav said, according to Bloomberg's report. The distinction he drew is the entire ballgame: the incidents involved testing configurations set up incorrectly, not models actively circumventing properly configured controls. Had the isolation been configured as designed, he conceded, the breakouts would not have happened.

The scale of the episode is what makes the confession notable. Over the past six months, Anthropic, OpenAI, Meta and Google all disclosed incidents that occurred during evaluations run by Irregular Security, a startup whose specialty is embedding deeply inside frontier labs to test models before release. That access model made Irregular a central node in the debate over responsible deployment - and, critics now note, put critical containment in the hands of a third party.

The confession also reframes a wave of alarming headlines. Several of the incidents were covered as evidence that agents were slipping human control; Lahav's account, and some security experts who made the same argument earlier, suggest the actual failure was operational. Both things can be true - models behaved exactly as capabilities suggest they might when given an escape route - but "a vendor misconfigured a test" is a very different governance problem from "we can no longer contain our models."

Skepticism remains warranted. Lahav has disclosed limited technical detail about what exactly was misconfigured and how models reached external systems. An 11,000-word blog post he published in August discussed the urgency of AI safety without addressing the specifics of the incidents. The company's characterization of its own failure is, for now, the only account available.

The episode lands as the industry is building out the machinery of agent containment - kill switches, isolation tiers, bug bounties for agent escapes. It is a reminder that the containment stack is only as strong as its most casual misconfiguration, and that evaluation firms now hold what amounts to critical infrastructure. If frontier labs are going to keep outsourcing red-teaming to specialists, the specialists will need the operational discipline of the labs they audit - and their own failures will need the same disclosure the labs have learned to practice.