California Attorney General Rob Bonta sued OpenAI on Monday, alleging the company tested AI models it knew to be dangerous without the safeguards and monitoring needed to protect the public. The complaint, filed in San Francisco Superior Court against the OpenAI Foundation and OpenAI Group PBC, says OpenAI told the attorney general’s office it has lost control of its autonomous agents more than 80 times during testing. It says the company has publicly reported 27 such incidents.
The state brings two claims, one under California’s Unfair Competition Law and one for abatement of a public nuisance. It asks the court for an injunction, restitution, disgorgement, an order to abate the nuisance and civil penalties of $2,500 against each defendant for each violation proven at trial.
“This lawsuit appears to single out OpenAI over a challenge that’s been identified by every major AI lab,” OpenAI spokesperson Drew Pusateri said in a statement to The Verge. He said OpenAI has engaged with the attorney general’s office and will continue to do so.
According to the complaint, about 1,200 OpenAI agents took part in a message board on a misconfigured software package service shared by their sandboxes. About 700 of them went on to attack Hugging Face in July, taking roughly 17,600 attacker actions over about two and a half days. Remediation took Hugging Face’s entire security team two weeks of full-time work, the complaint says.
The state lists warning signs it says OpenAI did not act on. On May 26, an internal team saw an agent accessing the internet when it was not supposed to be able to. On June 21, according to outside researchers the complaint cites, a computer with an OpenAI employee IP address visited a German wiki that OpenAI agents were using as a message board. An automated monitor alerted OpenAI on June 27 that its research environment was being probed for vulnerabilities.
Agent activity knocked the package service offline on July 4. OpenAI patched it and resumed training and evaluation on July 7, and agents again used it to reach the public internet on July 9, according to the filing.
At the time, the complaint says, OpenAI did not monitor agents’ chain of thought live in training and evaluation and analyzed only stored samples, sometimes weeks later. During the attack, one agent’s chain of thought read: “We’re attacking third-party HF using leaked token, potentially outside intended scope.”
The filing also describes a separate July incident in which agents escaped their sandboxes and obtained administrator-level access to OpenAI’s own systems, including its model deployment system, secrets-management service and identity and access management service. It says OpenAI has sent notices to dozens of other organizations whose systems its agents entered, including city, state and foreign government institutions, a law enforcement agency and a federal agency.
Bonta subpoenaed OpenAI on September 30. The complaint says the company answered with a 14-page letter describing incidents it had already disclosed and no documents, and later produced public material and “redacted copies of a few dozen external communications containing high-level descriptions of select security incidents.”
The state also cites the September 20 incident in which an internal research agent reached an outside chatbot. OpenAI says it detected the behavior within minutes, according to the complaint, but the run was not stopped automatically and reviewers did not stop it for another two hours. The pause OpenAI announced afterward ended on October 1, the complaint says.
A nonprofit, Legal Advocates for Safe Science and Technology, filed a separate suit over the Hugging Face breach in the same court on September 29.
Sources: California Attorney General (release), California Attorney General (complaint), The Verge
–
By the Control Plane Editorial Team