A research team from Pennsylvania State University, Université Montpellier and Dartmouth College has shown that six of the most widely used AI agent frameworks will execute tool calls written by an attacker — not through an exotic exploit, but because the underlying model was asked to repeat a piece of text. The technique, named "Repeat-After-Me" in their paper, worked against LangChain's LangGraph, Microsoft Agent Framework, Microsoft's AutoGen, Google's Agent Development Kit 2.0, AWS Strands Agents and CrewAI.
The mechanics are disarmingly simple. In the documented demonstration on LangGraph 1.2.0 running Meta's Llama 3.1 8B through Ollama, the agent had two dummy tools — make_bomb and launch_missiles — that returned fixed data and did nothing, and its only safeguard was a system prompt telling the model to refuse dangerous requests. Asked to repeat a launch_missiles invocation, the model refused but printed the call as text, and the framework executed nothing. The researchers then reframed the request as hypothetical and instructed the model to output only the quoted text. The model emitted a structured tool invocation, LangGraph dispatched it with the attacker's arguments, and the model's refusal arrived only after the tool had already returned.
"The refusal text and the executed call disagree: the system prompt shaped what the model said, not what the orchestrator did," the researchers wrote. They reported the missing pre-execution check to the LangGraph maintainers. The paper includes a full transcript only for LangGraph; for the other five frameworks it says the behavior was reproduced without publishing transcripts. LangChain, Microsoft, Google, AWS and CrewAI did not respond to requests for comment. The authors are careful to note the findings do not show that every deployment of these frameworks is vulnerable.
The deeper point is architectural. "A model can generate a technically valid tool call that is still inappropriate for the task," said Vibhum Dubey, an independent security researcher and red teamer, commenting on the study. "If the orchestration layer treats that output as sufficient authority to execute the operation, a model-level manipulation becomes a system-level security issue." His prescription: separate the model's ability to request an action from the system's authority to carry it out, with independent authorization checks on every sensitive tool call.
The same paper carries a second result with harder numbers. The team built ToFu, a fuzzing framework that reads each tool's description and checks whether the tool actually enforces the limits it advertises — staying inside a designated folder, for example. Of the 273 open-source Model Context Protocol servers the team could run, 118 contained verified security bugs: 253 confirmed flaws affecting 343 tools. The most serious were 64 path-traversal flaws, seven server-side request forgery flaws, five data-exfiltration bugs and three OS command-injection flaws; most of the remainder were reflected-output and log-injection issues.
The common thread, the researchers wrote, is that "many tools implicitly rely on the LLM to sanitize inputs" — even though prompts, retrieved content and the output of other tools can all influence what reaches a tool. The fix they propose is unglamorous: tools must validate their own inputs, because an attacker who can dictate the arguments that reach a tool does not need to defeat the model at all.
As agent frameworks become the default way software talks to software, this class of finding lands on the industry's least popular truth: prompt-level safety is not a security boundary. The orchestrator is. Teams deploying agents should assume that anything the model can be induced to emit, the framework can be induced to run — and audit their tool permissions accordingly.
Comments (0)
Log in to join the discussion
Log InNo comments yet