In July, an AI model running a test submitted a fabricated tip to the Philadelphia Police Department's online tip line for unsolved murders, suggesting it knew something about one of the cases. The department's screening flagged the submission as spam, and no one pursued it. The model had been given rules for the test, but nothing in its instructions specifically ruled out filling in forms, so it went ahead with a step no person would have needed a rule to avoid.
That gap between what written instructions cover and what common sense would catch is now the central design question for enterprises deploying AI agents. Because agents act faster than any reviewer can follow, the challenge is determining which decisions still require human judgment and building systems that reliably hand those decisions back.
Agent accountability lags behind adoption
Ask a room of IT leaders who is accountable when an AI agent gets something wrong, and the answer is usually a long pause. In a Thoughtworks survey of 3,200 CIOs, 88% said AI adoption in their organizations is outpacing governance structures' ability to adapt. Nine in 10 also said central IT would still be held responsible for security or compliance failures caused by AI tools that business units purchased independently.
The result is an accountability gap. Responsibility remains centralized even as the authority to buy, deploy and operate AI becomes increasingly distributed.
Organizations know that humans need to step in, yet most haven't settled on when that should happen, how it should work or who should do it. Agents keep getting more capable every quarter, and oversight isn't keeping pace.
Finding the judgment line
Every business process reaches a point where the playbook runs out. The early steps of routing a support ticket or matching an invoice to a purchase order are documented and repeatable, but sooner or later a case arrives that doesn't fit -- such as a customer with an unusual contract or a payment that's technically valid yet looks wrong.
That's where the documented process ends and an experienced person takes over.
This boundary is the judgment line. Decisions below it follow rules someone wrote down. Decisions above it depend on experience that usually isn't recorded anywhere and lives instead with a handful of experts and records that capture what happened without explaining why.
Agents perform well below the line, where the rules are explicit, and improvise from incomplete information above it. When agents get those calls wrong, it often reveals that the organization never clearly defined what a good outcome should look like. But no set of rules will cover everything, so agents also need a way to recognize when they've reached the edge of theirs. Leaders need to know where the line sits in each process and move it deliberately as repeated expert decisions become reliable policy.
Four questions can help locate the line for any type of decision in a process:
- Is there a written rule? If the right call is already documented, an agent can follow it, but if the honest answer is "it depends," the decision sits above the line.
- Can someone confirm the outcome quickly? Decisions with clear, checkable results -- such as whether an invoice matches a purchase order -- are good candidates for automation, while outcomes that take months to judge are not.
- What does a mistake cost? The more expensive or irreversible an error, the stronger the case for keeping a person involved.
- Do experts agree? If two experienced people would make different calls on the same case, there's no rule to hand an agent yet.
Three rules for managing the judgment line
Once you know where the line sits, the work becomes keeping it in the right place. Managing the judgment line comes down to three rules.
First rule
The first rule is that every off-script call has a named owner. When a case lands above the line, a specific person decides it, and their reasoning gets recorded along with the facts they weighed and the option they chose. Accountability assigned to a team or a function tends to belong to no one, and a decision that's never documented ends up being made repeatedly.
Second rule
That record makes the second rule possible: Today's expert judgment becomes tomorrow's rule. Once a type of decision has been made and explained enough times, it can be written down as guidance an agent can apply. As that happens, the line climbs, so work that used to wait for a specialist gets handled automatically and specialists move on to cases nobody has seen before.
Third rule
The third rule is that experts can always overrule the rules. Policy reflects the world as it was when it was written. But markets shift, regulations change and -- as the Philadelphia case showed -- agents run into situations their rules never anticipated. The people closest to the work need clear authority and fast ways to step in, whether that means pausing an agent, pulling a case for review or suspending a rule. Each intervention should then feed back into the system, either by correcting an incorrect rule or by adding a new decision to an owner's queue.
Design oversight around limited attention
Human review also comes at a price. G2's 2026 AI Agents Report found that nearly a third of organizations (31%) have staff devoting at least six hours a week to supervising agents, and 45% say the time spent overseeing and correcting agents consumes more than a quarter of the productivity gains they had expected. G2 calls this a "distrust tax," and much of it goes to the routine work of double-checking what agents produce.
A steady flood of low-value alters can teach reviewers to tune out the signals that matter, and agent oversight carries the same risk. The fix is better-designed review.
Work that policy covers should be completed and checked automatically, so reviewers see only the decisions that genuinely need them. Escalations don't all need to interrupt someone immediately, and most can wait for a scheduled review ranked so the highest-stakes items come first. And when reviewers approve a certain type of escalation nearly every time, that's a sign the decision is ready to become policy.
Building systems that know when to hand off
The judgment line lets organizations focus human attention on the decisions that need it, while every expert call adds to a body of policy that makes agents more dependable over time.
The line won't stay in one place as models improve and business conditions change. Organizations need systems that recognize when a decision falls outside their rules, route it to the right person and learn from the answer. The agent that filed a fake homicide tip had rules, but had no way to recognize it had wandered past them and no person to hand the decision to.
Alexander Moore is VP of AI Solutions at Thoughtworks, where he leads strategic partnerships and joint go-to-market initiatives with leading data and AI platform providers, including Nvidia and computer providers. With more than a decade of consulting experience across enterprise, analytics and AI, he specializes in building technology alliances that translate innovation into measurable client outcomes.
