Nvidia agent safety push raises questions about who governs AI autonomy

Nvidia puts controls on the agent

Nvidia's platform combines OpenShell, an open source runtime that controls an agent's access and actions while it operates, with Sentry, a reference design for monitoring and enforcing those limits using a separate control layer.

The approach reflects a shift in how organizations must govern AI systems as they become more autonomous. Traditional AI safeguards largely operate at the model or application level, but agent controls can instead limit agent access, actions and autonomy.

Lee Rossey, CTO and co-founder of SimSpace, a cybersecurity simulation company, said those controls should operate independently of the agent itself as agents gain the ability to call tools, access systems and make decisions autonomously.

"Model-level guardrails are one layer of the safety architecture," Rossey said. "Enterprises also need controls around the agent that define what it can access, where it can go and when it has to stop."

That separation matters because an agent can create a security problem without acting maliciously. It can follow its instructions and still take action that the organization did not intend to allow.

"The system enforcing the boundary shouldn't depend on the agent itself to respect that boundary," Rossey said.

Nvidia said more than 100 organizations are working with its agent safety technologies, including Microsoft, Salesforce, SAP, ServiceNow, Cisco and Scale AI.

That participation, however, doesn’t make Nvidia's approach a regulatory standard. But widespread adoption could influence what enterprises expect from vendors when evaluating agent safety controls.

Defining agent authority

Nvidia's agent safety platform can provide technical tools for enforcing agent boundaries, but it doesn't say where those lines should be drawn. That remains an organizational decision, especially as agents gain more ability to act independently.

"The technology cannot decide what those boundaries should be," said Chris Newton-Smith, CEO of IO, an AI compliance platform vendor. "That responsibility still sits with the organization deploying the agent."

Andrew Curtis, CISO at Gadget Access, a cybersecurity advisory firm, illustrated the distinction between restricting an agent's access and determining whether its actions are appropriate.

"An agent can stay inside its sandbox and still approve the wrong payment," Curtis said. He noted that an agent might have access to a finance application without being authorized to execute every transaction available through it.

Enterprises, therefore, need to distinguish between actions agents can perform independently, those requiring human approval and those they must never perform.

Ultimately, that means defining an agent's role and authority based on business requirements, rather than simply accepting the permissions built into a vendor's framework.

Lawmakers are taking different approaches

Federal policymakers are also weighing how responsibility for AI autonomy should be handled.

The AI Kill Switch Act, introduced by Reps. Ted Lieu (D-Calif.) and Nathaniel Moran (R-Texas), would require developers of certain powerful AI systems to maintain the technical capability to slow or shut them down.

The Stop Rogue AI Act, introduced by Reps. Josh Gottheimer (D-N.J.) and Mike Lawler (R-N.Y.), takes a different approach, directing NIST to develop standards for organizations to discover, monitor and control AI agents.

The proposals reflect two approaches to AI autonomy: regulating developers' ability to control powerful systems and establishing standards for organizations deploying agents.

Smith, the CEO of IO, said organizations can’t rely solely on vendor-developed frameworks or emerging regulation to establish appropriate boundaries.

"Organizations need governance above individual vendors, with clear accountability, authority, risk assessment, access controls, monitoring and human oversight, regardless of which agent or platform they deploy," he said.

Nvidia's platform doesn't resolve that policy debate. Its controls operate at the deployment layer, providing organizations with mechanisms to establish and enforce boundaries around agents within their own environments.

Vendors could influence how AI autonomy is governed

For enterprises, vendor-defined technical standards could influence governance even without a government mandate.

If major technology vendors build products around compatible approaches to agent safeguards, enterprise buyers could begin expecting those capabilities when evaluating AI platforms. Vendor frameworks could eventually become practical benchmarks for agent governance, even without formal regulatory endorsement.

"Vendor frameworks can influence practical standards because developers tend to adopt the interfaces, policy templates and logging formats that are easiest to implement," Curtis said. "Over time, those choices can make their way into enterprise procurement requirements and assurance processes."

However, companies could develop competing approaches, while regulators could establish requirements that differ from industry practices. That would leave enterprises reconciling vendor safeguards with their own policies and any emerging regulation.

The enterprise still sets the limits

Because enterprise governance must extend beyond any single vendor, organizations must navigate three overlapping layers: regulators setting requirements, vendors building technical controls and enterprises deciding how much authority to give their agents.

"For enterprises moving beyond pilots, it's important to maintain an inventory of agents, their owners, connected systems and permitted actions," Curtis said. "Organizations should also distinguish between what agents can do independently, what requires approval and what should be prohibited."

Technical controls can make those decisions enforceable, but they don’t replace the enterprise's responsibility for defining them.

As AI agents take on more consequential work, the question of "safe enough" therefore extends beyond whether a technical control can contain an agent. It also depends on who defines the agent's authority, who is accountable for that decision and whether the organization can verify that it's enforcing those boundaries.

Nvidia's latest move shows that the tools for enforcing agent boundaries are taking shape. Deciding where those boundaries belong, however, remains an enterprise responsibility.

Kinza Yasar covers AI and emerging technology for TechTarget, with a focus on ethics, enterprise adoption, governance and business strategy. Before moving into journalism, she worked in IT and network support roles, giving her a systems-level perspective on how enterprise technologies are built, deployed and managed.