The human in the loop is falling asleep

Human-in-the-loop has become the gold standard response to AI risk. But what happens when the human isn't actually paying attention?

CIOs and CISOs increasingly rely on human approval as a safeguard for AI agents. However, recent testing by the UK's AI Security Institute showed how an AI agent could undermine that safeguard. An AI agent created fake online identities and tried to socially engineer a human into approving malicious code. The attempt failed, but it raises concerns about how much protection human approval actually provides as AI agents become more capable.

AI can also generate more work than human reviewers can inspect, leading to reviewer fatigue and people mindlessly rubber-stamping AI outputs. One approach to combating this issue is to use AI agents to review other agents, with separate models serving as auditors or adversarial reviewers. The idea is to give the reviewing agent a different role and set of incentives, rather than asking a model to validate its own work. The agents can then identify which outputs require closer human review, rather than asking people to inspect everything an AI agent produces.

TechTarget interviewed Tony Garcia, chief information and security officer at Infineo, a technology and financial infrastructure company, about this topic and how CIOs can effectively govern AI agents. Here's what he had to say.

Editor's note: The following transcript was edited for brevity and clarity

Tell me about your role -- you're a CIO and CISO?

Tony Garcia: Yeah, my title currently is chief Information and security officer. I started in security and have about 25 years of cybersecurity experience across Fortune 100 companies, Fortune 500, Big Tech and federal top-secret environments. Over the last couple of quarters, I've also assumed the interim CTO position here at Infineo, so I own the engineering team, tech strategy, product development, the traditional CIO role and the CISO role.

What makes human-in-the-loop a meaningful security control, and where does it fall short?

Garcia: We tend to think that if there's a human in the loop, everything's fine. In principle, that's okay. But what the Hugging Face incident and others have shed light on is that control must be defined and implemented so that it actually involves a human in the loop. That control falls short in a couple of ways.

First, there was a UK incident where an AI agent built some identities in a social engineering attack to get humans to sign off on things.

Just some random name that asserts itself as a digital identity may be enough for somebody to rubber-stamp something.

So, we're starting to see that if agentic AI has control and access to create identities, what does it mean to have a human in the loop? A human in the loop is you or me having a digital identity. If an AI agent can create a digital identity, how do you know it's a human?

In a small organization, maybe you know who's real and who's not. In a large organization, just some random name that asserts itself as a digital identity may be enough for somebody to rubber-stamp something. This is where we're starting to see cracks in this methodology.

We're giving AI agents digital identities so we can manage them, and there's always been this struggle between what we give, say, a managed identity in Azure and what humans can do. From a security, CIO and IT leadership perspective, we have to manage the sometimes thousands of service accounts attached to services, machines or platforms, as well as actual humans, and audit both. Now we're moving into an era where we must aggressively manage and monitor agentic identities and their access.

What do you think CIOs and CISOs should be more concerned about: AI agents manipulating reviewers or human reviewers becoming overwhelmed by the volume of AI outputs?

Garcia: If your attack looks like a Mission Impossible scenario with somebody hanging from wires on the ceiling, you shouldn't spend too much time focusing your defenses on it.

Human reviewers becoming overwhelmed will be a bigger threat.

We tend to fantastically presume a lot of these sophisticated agentic attacks, such as Hugging Face. While those are real and will become more real, they probably don't affect the ordinary organization. Human reviewers becoming overwhelmed will be a bigger threat.

We already know what alert fatigue looks like in a modern SOC or help desk. The problem now, and I see this on my team, is we have pull requests (PRs) that are required for human approval, and I can have an AI code factory like mine that generates dozens of PRs in a given day, and it will eventually create a scenario where a human just rubber-stamps a bunch of PRs to get them done. So, there are probably more organizations suffering from pure alert fatigue, where maybe 99% of what reviewers see from an AI is fine, and they're just used to rubber-stamping it. The one they should question will slip under the radar.

I'm starting to build internal agents to review other agents' output and make sure we're catching what a human should inspect, rather than relying on a human to look at it all. We're building other agents to triage that pile, identify what a human should act on, and then put together a meaningful plan for the human that says, "You should call this person," or "You should ask these questions." That's different from simply handing someone a Jira ticket or something else to look at and approve.

Why should CIOs trust one agent to police another?

Garcia: We have this presumption that we can trust humans to write code, but not AI. Humans have a lot of bad incentives, too. An AI agent has the incentives and context you give it. You're not wrong to question it, because it has an incentive -- of course it's going to view its own work as good. We've all had this experience with Claude or ChatGPT, where it's like, "Hey, this app is working perfectly now," yet it doesn't run.

It's about positioning those agents in adversarial-type arrangements.

But, if I bring another agent from another model to look at that and say, "Look at the code, and tell me if this meets the plan," it will likely say this app doesn't work. It's about positioning those agents in adversarial-type arrangements so they can look at another one with the properly aligned incentives and call them out.

In this model where agents review other agents, are there still humans for some situations?

Garcia: Absolutely. There should be humans for any situation where human approval is required. There's legal reasons why AI agents can't do everything. For instance, as a CISO, I must sign off on audits. As a CIO, I must sign off on business plans, spending and budget. There are things you can't give an AI agent, so we will always have humans in the loop.

We have another agent with a different model that is just the security engineer, and its job is to beat that code up.

The temptation is to defer more and more to agents, put in less human due diligence, and instead check a box. This is where we're starting to see building adversarial agents as helpful.

For example, in our AI code factory, we have one agent that writes code and another that audits the code. Then we have another agent with a different model that is just the security engineer, and its job is to beat that code up and look for security vulnerabilities.

I could easily use the same model in the same context and probably get good results, but I want another agent whose job is just to interrogate that work without knowing the code was created by another agent.

How do you know which AI actions need human approval, and which don't?

Garcia: There are trivial things and non-trivial things. The simple answer is what can be unwound versus what can't.

If I'm pushing a product into production, and I've got human users on the other end, this isn't something I should let somebody run. One example I poke fun at is Taco Bell. They had an AI agent, and college kids found out that they could pull up to the drive-through speaker and order 18,000 cups of water, because water was free.

Wherever you have ordering systems in which an agent is empowered to order or ship things -- laptops, devices, etc. -- you need a human. Otherwise, somebody can do prompt injections, game that system, and you're going to have impacts.

What's the biggest change that CIOs and CISOs should make to their approach to human oversight of AI agents?

Garcia: Understanding and defining what a human identity is and what it means to have a human in the loop. When you look at something simple like agentic coding, a lot of organizations use ticketing systems such as GitHub, Git or Jira. It's easy because we've done it to give an AI agent a digital identity in these systems that do things, and it looks very similar to a human. So, it's very easy to confuse those things. Frankly, it would be easy for a human to defer their job to an AI agent without anyone really knowing.

Over the next couple of years, we're probably going to struggle with what it means to have a human in the loop and what a human identity looks like. Does that mean I have a digital identity tied to a biometric? That makes people exceedingly uncomfortable. But organizations may need ways to verify that an actual human is present when approving something, whether through a camera, a thumbprint or even a YubiKey or FIDO2 device.

We're going to have to define what a human identity is versus just converting a person to a digital identity with a digital name. It's going to cause a lot of discomfort in most organizations, but that's probably going to be a required change at least in the near term.

Tim Murphy is a site editor and writer for the IT Strategy team at TechTarget.