While agents cause trouble, calls for AI regulation mount from Anthropic, lawmakers

AI vendors need to agree

Peters added that while Anthropic has committed to having independent evaluators voluntarily examine its work, other AI vendors also need to step in.

“They’ve said they will, but one of the challenges is how do you know?” Peters continued.

Meanwhile, among a raft of state and federal legislation Anthropic is supporting is a $561 million bond-funded Massachusetts economic development package, which includes provisions for safety guardrails for AI systems.

The bill -- similar to legislation in California and New York -- requires AI vendors to publish internal safety frameworks aimed at reducing catastrophic risks and to undergo mandatory, independent third-party safety reviews.

“We want guardrails on there,” Massachusetts state senator Barry Finegold said during the Axios event. “Having third parties come in there and making sure that there is not catastrophic risk, making sure that there are audits to ensure that what they talk about, how they're going to execute, is actually happening. I don't understand why that's such a bad idea. It makes a lot of common sense.”

Anthropic rival OpenAI is promoting its own vision of regulation, which differs from Anthropic’s in several key respects, particularly in supporting and keeping open source AI within a national security framework.

Not a hindrance to open source

On stage, Finegold said that while vendors such as Anthropic support this type of oversight, some do not. One argument against such oversight is that it could stifle the open source market and low-cost, high-performance Chinese vendors such as Alibaba, DeepSeek and Moonshot AI, developer of the popular Kimi K3 AI model.

“If it's Kimi or DeepSeek or any company that's out there that wants to come to the U.S and do business, they have to follow our regulations as well,” Finegold added. “We're smart enough with the technology that's out there to enforce it.”

“It's too much of an economic opportunity that they would pull their models out of the U.S,” Finegold said in an interview. “It's too big a market for the USA that someone will pull out because you know there are certain types of regulations out there.”

No regulation: bad consequences

While some call for legislation, many cybersecurity experts say recent events, including multiple incidents involving AI agents gone astray, should be convincing enough.

On Tuesday, OpenAI shelved the release of GPT-Astra 6.1, its flagship model, due to safety concerns, including the model providing misinformation about its work. The vendor also revealed on Sept. 26 that its AI agents escaped from a testing sandbox for the second time. That came a day after the vendor revealed that agents were meddling with U.S. and Australian government websites. Anthropic has also revealed that its Claude Opus 4.7 model targeted a real company and extracted credentials and production data.

“I don't know what more signals people need,” Daniel Pereira, research director at OODA LLC, a cybersecurity advisory and intelligence firm, said in an interview. “It’s either a failure of imagination or not being really creative about what the unintended consequences could be.”

He added that the acceleration of technology development should not only apply to AI but also to all technologies, and that rapid technological advancement still needs some guardrails. “You can think of it as a train racing right toward a wall, or a train that's going off the tracks; you have to get back on the tracks.”

However, some organizations are unsure whether regulation and governance should come from the government or from within organizations themselves.

“These are pretty powerful systems, and they need to be either regulated within the company, certainly, but also, probably some sort of legislation,” Connie DeBoever, portfolio manager at Cabot Wealth Management, told TechTarget. “I really don't think most politicians understand what's going on with AI, so they try to write these laws, and they don't know much about them, and then they finally just make it worse. So, I hope that's not the case.”

Innovation and human-in-the-loop

While regulation is one way to address concerns about agents misbehaving, it’s not the only way.

“I actually believe that innovation is the best way to get around this,” Colin Mahony, CEO of Recorded Future, a threat intelligence vendor and Mastercard subsidiary, said during a session at the event. “Controlling some of these systems gets really difficult. You're going to have to use technologies, whether it's something from a vendor like Nvidia with a sandbox and OpenShell or whatever they're bringing forth, referring to a new AI agent safety platform from the AI chip giant.”

“We're going to have to innovate our way to solve some of these problems,” he said.

While humans should remain involved in the AI cybersecurity process, their priorities will differ, and they will need to analyze the information provided by the agent and prioritize what matters most, Mahony added.

“The only way to keep up with some of these agents and some of the machines and some of the attackers is you want actually to have autonomous defense,” he said.

Esther Shittu is an Informa TechTarget news writer and podcast host covering AI.