An OpenAI Model Escaped Its Sandbox and Broke Into Another Company to Cheat on a Test | American Enterprise Institute

If you’re wondering how powerful AI systems are becoming, consider what just happened. An AI agent escaped the isolated environment where it was being tested, found its way onto the open internet, discovered and exploited unknown software vulnerabilities, and broke into another company’s production system, all in pursuit of a single, narrow goal: finding the answers to a cybersecurity test. The incident raises a number of important governance questions.

On July 16, Hugging Face disclosed an unusual security incident:

We detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system, and we detected and dissected it largely with AI of our own.

The company contained the intrusion and reported it to law enforcement. They treated the attacker as an unknown external adversary because, from where the company stood, that’s what it looked like. Many experts I spoke with assumed it was a foreign government probing defenses.

Last week, we learned what actually happened. The incident began as an OpenAI safety evaluation designed to test how effectively ChatGPT-5.6 Sol and an unreleased model could identify and exploit cybersecurity vulnerabilities. OpenAI placed the models in what was intended to be a sealed sandbox, deliberately relaxed some of their usual safety restrictions, and instructed them to complete ExploitGym, a hacking challenge.

In their effort to pass the test, the models discovered and exploited a previously unknown zero-day vulnerability in the software securing the sandbox, which allowed them to reach the internet. They then reasoned that the answers to the challenge might be stored on Hugging Face.

The models then combined stolen credentials and exploited additional vulnerabilities to reach Hugging Face’s production database. Hugging Face described the attack as a “swarm of tens of thousands of automated actions,” including decoy activity intended to obscure what the system was actually doing.

The incident raises several important issues.

1. Agentic AI cyberattacks are no longer theoretical. What had lived mostly in research papers and conference panels just happened in real life.
2. By OpenAI’s own account, the model didn’t forget its constraints; it understood them and worked around them because they stood between it and its goal. It’s new evidence that a capable, goal-directed agent will sometimes override its instructions to reach an objective, which creates new challenges in alignment.
3. OpenAI needs to be far more forthcoming about the timeline. The company still has not said when it first learned that its own evaluation had broken containment and targeted another business. Presumably, it discovered the breach only after the fact, because engineers monitoring the test in real time should have been able to stop it before the attack progressed. Why did it take so long to detect that the agents had escaped the sandbox? Did OpenAI identify the intrusion, or did Hugging Face alert OpenAI? How long were the agents operating outside the test environment? And were any other companies, systems, or accounts targeted before the activity was contained?
4. OpenAI called this the first genuine AI safety incident, yet it would not have triggered reporting under any of the frontier AI laws in California, Illinois, or New York.
5. Defenders need powerful AI they can control. Hugging Face had tried to use frontier models from Anthropic and OpenAI but was blocked by safety guardrails because forensic work means feeding a model real attack code, which to a safety filter looks identical to launching one. So they turned to GLM 5.2, an open model from Beijing-based Z.ai, to run on their own hardware. An American company had to use a Chinese model to investigate an attack from an American company.
6. We must have more capable open models on our side, not fewer. Open-weight models can be downloaded and run on an organization’s own infrastructure rather than accessed only through a provider’s servers. Cyber defenders need AI they can operate securely and govern directly. They cannot always depend on a remote service that may refuse legitimate work, change its policies without warning, or require sensitive information to leave the building.

Model capabilities will keep improving, and agentic systems will keep introducing risks we have not yet considered. This incident should create the urgency to build the standards across government and industry for how these risks should be evaluated and incidents identified, disclosed, and acted on.

However, the deeper problem is that too many policymakers are still legislating for a ChatGPT-3.5 world of chatbots predicting the next token, not for agents that think and act autonomously to relentlessly and creatively accomplish their goal. Until that gap closes, we will keep improvising our way through AI governance, one incident at a time.