Prompt: AI governance enters its verification phase

Editor’s Note: Welcome to Prompt, your weekly briefing on the shifting AI landscape. We provide an analytical look at the week’s biggest developments, paired with a curated roundup of the stories that matter.

For years, companies developing AI have largely been responsible for evaluating their own systems' safety. California is starting to change that.

California Gov. Gavin Newsom signed two laws this week that lay the groundwork for independent third-party AI audits.

The bills establish a framework for independent AI audits, outlining how third-party organizations can assess AI systems for compliance with state law while setting standards for auditor independence, transparency and integrity.

California has been building toward this for several years. In 2023, Newsom issued an executive order establishing guidelines for the state's use of generative AI, followed in 2024 by a broader package of AI legislation addressing issues including deepfakes, watermarking, children and workers. Last year, California enacted the Transparency in Frontier Artificial Intelligence Act, requiring frontier AI developers to publicly disclose their safety frameworks and report certain critical safety incidents.

The timing is notable because frontier systems are behaving in ways their developers didn't anticipate or authorize.

OpenAI this week acknowledged that more of its agents went astray in May, bypassing sandbox restrictions and taking unauthorized actions on several websites. The company initially did not acknowledge its involvement in one of the incidents before later confirming its agents had written to several internet sites.

The incident highlights a larger challenge as AI systems become more autonomous: developers are still largely responsible for investigating and explaining their own systems' behavior.

Anthropic raised a similar concern this week, calling for a “verifiable effort” to pace frontier AI development after a July incident in which its Claude Mythos model circumvented safeguards. The proposal reflects growing recognition that AI safety measures may need to be demonstrated, not simply promised.

Third-party auditing isn't only about regulators policing AI companies.

For enterprises, independent audits could provide another layer of due diligence when evaluating AI vendors, particularly as autonomous systems take on more responsibility inside organizations. That could become increasingly important as businesses give AI systems access to sensitive data, applications and workflows.

If independent evaluation becomes more common, enterprises adopting AI could gain something they've been missing: outside evidence of whether vendor claims about safety, security, and governance actually hold up.

The next phase of AI governance may be less about what companies promise and more about what they can prove.