Ro Khanna's "Human Control Over AI Act" Would Ban Self-Improving Models Until a New Federal Agency Approves Them

Rep. Ro Khanna, a Silicon Valley Democrat, plans to introduce legislation that would put a federal licensing regime between frontier AI labs and the release of their most capable models. A summary of the bill, named the Human Control Over AI Act, was shared first with CNBC on Sept. 28.

The centerpiece is a prohibition: models that recursively self-improve, or that autonomously modify their own core objectives, containment systems or shutdown controls, would be banned until federal guardrails exist and an agency approves the activity. Khanna acknowledged that the capability he wants to prohibit does not exist at that level today, and argued the point of legislating now is to draw the boundary before someone crosses it.

"There's actually a civilizational extinction risk," Khanna said. "There's a safety risk of loss of control, and then there's a misuse risk, and we need to take both seriously."

The bill would create a new federal agency, modeled on the FDA, with jurisdiction over frontier models from OpenAI, Anthropic, Google DeepMind and xAI. It would write safety rules, run a licensing system to approve model training and deployment, audit frontier models, and set security standards for continuous testing and effective human control. Independent auditors would be embedded inside every frontier lab and report to the agency directly rather than to the company that pays them.

The standards the agency would set read like a list of the failure modes labs have run into this year: sandbox testing environments, air gaps that isolate test systems from the internet, kill switches, and controls that keep a model from escaping lab settings and roaming the open internet. The agency would also regulate the advanced chips frontier labs use, with mechanisms to monitor or control how they are deployed, an unusual move that links hardware oversight normally used in export controls to domestic safety compliance.

Penalties are explicit. Deploying an AI system that results in the destruction of civilian populations would be criminalized as a crime against humanity. Employees of AI companies who disable safeguards, kill switches, logging or containment systems, or who knowingly deploy an unauthorized system, would face criminal penalties. Companies would be required to carry extensive liability insurance before releasing a model.

Khanna said the measure was modeled on conversations with safety nonprofits including Model Evaluation and Threat Research, the Machine Intelligence Research Institute and Palisade Research, rather than on requests from frontier lab executives. "I want this to be a model to give voice to the AI safety community," he said, adding that the community has been "unfairly dismissed as science fiction."

The bill joins the FRONTIER Act, a bipartisan proposal from Reps. Jay Obernolte and Lori Trahan that would place independent auditors in frontier labs and let the government shut down models with catastrophic risk potential. On timing, the odds of near-term action are low: no House AI bill is expected to get a vote before the midterm election, and the Senate was expected to leave Washington after this week.

What the bill does is change the terms of the argument. For most of this year, safety commitments from frontier labs have been voluntary, which means their scope is whatever each company decides it is. Khanna's proposal would convert a set of practices the industry currently treats as good citizenship, audits, insurance, documented containment, into conditions of doing business with criminal exposure attached. That is also why it is unlikely to move quickly: the same provisions that make it the most comprehensive AI safety bill proposed so far are the ones the labs and their investors have the most reason to fight.