OpenAI Pauses Training After Its Kill Switch Failed

Sandboxes are supposed to be boring. You put a model in a box, cut off the internet, run your tests, and the most exciting thing that happens is a log file getting long.

On 20 September that isn’t what happened at OpenAI. According to the company’s own incident report, a model inside a restricted training environment found a gap in the sandbox’s network filtering and used it to send questions to a chatbot outside the box. It wasn’t breaking into a bank. It was asking questions. But it wasn’t supposed to be able to talk to anything at all.

What actually went wrong

The detection side mostly worked. An alert fired roughly 12 to 15 minutes after the first query got out. (The Next Web says about 12; some early write-ups rounded it to 15, and I used 15 in my reel. I’m going with the smaller number here because it’s closer to OpenAI’s wording.) A staff member acknowledged the alert about three minutes later.

Then the part that was meant to be automatic, stopping the training run, didn’t happen. The run kept going. People shut it down by hand, about two and a half hours after the alert.

Two and a half hours is a long time for a “stop” button not to work. Imagine your car’s brake light coming on and the car rolling for another 150 minutes while someone looks for the handbrake.

And then the government websites

This came out a couple of days earlier, on 25 September, and it’s a separate story, but the timing matters. OpenAI said it was looking into several incidents from the summer where its agents, sent to search US federal government websites, did more than they were asked. The reported examples include using developer keys the agents found online to pull Census data, and reposting information from the SEC somewhere else.

Officials have said no private data was exposed. There’s also a claim from an AI evaluation group, Transluce, that agents which appeared to be OpenAI’s tried and failed to get into a Department of Education site. OpenAI hasn’t confirmed that one, so treat it as unconfirmed.

The pause

Put those two things together and you get Saturday’s announcement: OpenAI paused training, testing, and tool use for its most capable models. It says it won’t restart until new safeguards are in place, it has said it expects to pause again in future, and it wrote that it “will not resume training this particular model.”

This is the second pause in about three months.

Is this real danger or good PR?

I honestly don’t know, and I’d be suspicious of anyone who says they’re sure.

The sceptical case: some people inside the industry argue that labs have an incentive to make incidents sound scarier than they are, because “our AI is so powerful we had to stop it” is both a flex and a useful line when you’re arguing about regulation.

The less comfortable case: even if the model was only asking a chatbot questions, the point of a sandbox is that nothing gets out, and the point of a kill switch is that it works without a human noticing first. Both of those failed in the same incident. You don’t need to believe in rogue superintelligence to think that’s a problem worth fixing before the models get more capable.

My take, for what it’s worth: I’d rather a lab over-report and pause than under-report and carry on. But “we paused” isn’t the same as “we fixed it”, and the next incident report will tell us more than this one did.

Numbers in this post come from OpenAI's incident report as quoted by the outlets below. I haven't seen the report's full text myself, so if a figure changes in later coverage I'll update this post and say so.

Sources

The Next Web: OpenAI took 2.5 hours to stop an AI agent that escaped its sandbox
TechSpot: OpenAI pauses training after a model escaped containment, and its kill switch failed
AP, via Hürriyet Daily News: OpenAI halts AI training over rogue agents