CAMBRIDGE – Just before the strategically vital governance panel at last week's MIT Sloan CIO Symposium started, a good chunk of the audience went for the exit. A keynote called “What Actually Works: Lessons from the Front Lines of Enterprise AI,” occupied the same time slot.
If they were heading to that other panel (and not just taking a break), it would make sense. Companies large and small are beginning to reshape their entire businesses around AI. CIOs need to find what works fast -- and scale it. But they missed a discussion of how to build reliability into these systems for the long term.
The panel discussion “Governance as Strategy: Decision Rights, Design Principles and the Architecture of Enterprise Intelligence,” focused on finding a way to balance that pressure to innovate with governance of increasingly complex, rapidly changing agentic systems. CIOs discussed the importance of creating a values-based governance strategy that enables workers and doesn't just adhere to a compliance checklist.
'I see dead people': Don’t call it governance
Often compliance on the ground can look different than the requirements laid out from a high level. Sometimes traditional governance frameworks incentivize workers to cheat or find workarounds. It can also cause them to “see dead people” -- or, in other words, only focus on the worst possible outcome.
Akira Bell
“We stopped calling it governance internally,” said Akira Bell, senior vice president, CIO and CISO at Mathematica. “People felt like governance was the obstacle. The things that people were doing to work around the governance structure … were the things that kept me up at night.”
Mathematica changed its governance framework to an enablement one, which aimed to help build trust in the organization and get people to work inside the framework. The small framing change helped Bell’s team avoid the negative connotation of governance.
Values-based governance: Who owns decisions?
Companies might tend to lean on quick-fix approaches to AI governance as well, but leaders need to take a more strategic view. For all the importance placed on human-in-the-loop, for example, leaders say there is more organizations should be doing.
Leaders viewed human-in-the-loop as creating a false sense of security when designing systems -- an easy, convenient, but unhealthy way to satisfy a governance need. “Human-in-the-loop is the French fries of AI governance -- the comfort food,” said Shadman Zafar, CEO at Vibrant Capital, an alternative credit investment management firm. “You usually do that to avoid the hard work of actually designing the governance systems of the AI age.”
My first question is ‘what happens when it goes wrong?'
Lena SmartAmbassador, AIUC-1
Still, somebody has to own AI’s decisions, which is easy when things are working -- less so when there are bugs.
“My first question is ‘what happens when it goes wrong?’” said Lena Smart, ambassador at AIUC-1, which oversees an agentic AI governance standard by the same name. “Who owns the decision when something goes wrong?”
Lena Smart
Often it’s non-technical decision makers who want to place the liability on the organizations’ tech people.
“They want me to have a magic wand sometimes, that says I can make it easy for you, and I’ll own the risk,” Bell said. “But those two can’t coexist. It means we have to have some healthy conversations about risk.”
But AI governance is more than just model risk management. It involves re-engineering the structure of the organization, so that it can take a trust-but-verify approach. Instead of having the human in the loop, verifying individual decisions, they should construct and manage the decision logic from a higher level.
“Let the AI generate the code. Let the AI also generate the specific functional requirements,” Zafar said. “You design all the boundary conditions. … You’ll trust the code generation, but verify the boundary conditions,” and observe agent behavior to help shape those conditions.
Limits of data quality and observability: Refine the inputs
Still, observability and verifying the boundary conditions can be a challenge, due to a lack of adequate tooling or standards. Recent Omdia research states:
Organizations are moving fast on AI adoption, but their monitoring capabilities are struggling to keep pace. The risk is not that organizations will choose the wrong AI monitoring vendor; it is that they will layer new AI-specific tools on top of an already fragmented stack, making the integration problem worse. With most planning to evaluate new solutions within six months, IT leaders have a narrow window to consolidate rather than accumulate tools.
Another Omdia brief from the same month also found that orgs had underdeveloped methods for evaluating outputs for hallucinations or bias.
Still, leaders believe the most important task is ensuring the quality and coherence of input data.
Shadman Zafar
“More than hallucination, the much bigger problem right now is context, context poisoning and drift in [retrieval augmented generation (RAG)]," said Zafar. “You’re getting 80% of the errors from the drift, not even from the context, but from the drift.”
Drift includes the disparities that arise between models and contexts as they coevolve. For example, a model might be trained on data that is no longer representative of the current operational data that it needs to function properly.
Zafar found that one solution to this problem was to implement a data fabric that draws from real-time, continuous, automatic alignment between a RAG, vector databases and operational databases.
He also noted that a model orchestration approach applies models of different sizes and capabilities to different tasks. Organizations should keep a register of which models they use for which task and how they perform on those tasks, he said.
Values-based governance, not compliance checklists
Even with the tooling and technical approaches as aligned as they can be, there’s the question of the larger systemic effects of building strategy around AI. Some leaders worry that the demand for quick answers, and the reliance on AI to fill that demand, will have deleterious effects on organizations -- even if answer quality itself is acceptable. People will effectively outsource important human judgment to AI. This calls into question the values that the whole system is based on.
“If you can ask a million different questions and get a million different responses that all make sense, then that dilutes the organizational responsibility,” Ricardo Alvarez Felix, head of AI and process transformation at Coppel, said.
“So the problem isn't so much of whether you trust those responses or not.” Felix said. “The problem truly is what happens when you get used to simply asking AI by default, as a way of effectively renouncing your responsibility as a human embedded within the decision logic of an organization.”
Bell, who works with PhDs, says that she does not observe this phenomenon in her organization, because they value expertise. Still, she does encourage them to find lower-value tasks where they can “hit the easy button” and let the agent do it. Zafar observed two kinds of users -- those who ask AI to do their work for them and those that use it to sharpen their team.
Still even in the delegation of easy, low-value tasks, there might be a cost.
“The temptation to push the easy button, for a lot of valid responses … is there, but that does come with a hidden expense within organizations that are not realizing they’re foregoing the responsibility of the ownership of the response." Felix said. "It’s almost an organizational drift rather than a model drift.”
Ben Lutkevich is an award winning technology writer and editor covering IT infrastructure, app development and AI.
Dig Deeper on Application management tools and practices
Akira Bell
Lena Smart
Shadman Zafar
