putilov_denis - stock.adobe.com
Your AI isn't failing -- your metrics are
AI doesn't know what matters to your organization. It only knows what the business measures -- and weak metrics become problems when AI treats them as the goal.
If AI is as powerful as everyone insists, why are so many organizations struggling to show a return on it?
Here is the uncomfortable answer: because AI optimizes exactly what leaders tell it to optimize.
Most organizations don't have an AI strategy problem; they have an AI ROI measurement problem. They have pointed an extraordinarily capable optimization engine at metrics that were built to track activity, not to define business value. The people closest to this problem are not the executives approving AI budgets but the data and analytics leaders who define, certify and serve the very metrics those systems will soon turn into operating instructions.
AI amplifies your incentives
AI has one defining behavior: it relentlessly pursues whatever objective you give it. That is its greatest strength and its greatest danger. This is economics, accelerated: incentives drive behavior, and AI acts on them faster and at greater scale than any workforce ever could. Every metric you reward becomes an instruction. Every optimization target is a statement of what the organization values.
This is the AI utility function -- the mathematical definition of "good" that guides every decision the system makes. Put more plainly: your metrics become AI's laws.
At heart, an organization isn't what it values. It's what it rewards. AI simply takes that literally. A human team hedges against a bad metric, using judgment, ignoring the directive that makes no sense. AI offers no such mercy.
3 ways this goes wrong
Optimize a call center for average handle time, and AI will shorten conversations. Calls get faster, but customers feel rushed, problems are partially solved and retention quietly erodes. You rewarded speed and paid for it with loyalty.
Optimize a clinic for patients seen per day, and AI tools designed to improve efficiency will push clinicians to shorten every encounter. Throughput rises, but quality of care falls, follow-ups get missed and outcomes suffer. You optimized volume at the expense of the only thing that matters -- whether patients get better.
Optimize a sales team for meetings booked, and AI will fill the calendar. Activity explodes, but revenue does not move because a booked meeting was never the goal. Closed business was. You optimized motion and mistook it for progress.
In each case, the technology performed flawlessly; leadership simply defined success incorrectly. This is Goodhart's Law, accelerated: once a metric becomes the goal, it stops measuring what the business actually cares about.
Efficiency is not value
All three examples share the same misconception: confusing efficiency with value. Efficiency asks, "Can we do this more quickly?" Economics asks, "Is this worth doing?" AI excels at reducing friction, but eliminating it from low-value tasks only leads to faster low-value work.
This is why so many AI ROI conversations feel hollow. MIT's Project NANDA reported in July 2025 that 95% of organizations have seen no measurable return from their GenAI investments. Not because the technology failed, but because they asked it to optimize activity rather than value.
Audit your KPIs before you automate them
This is where the problem becomes a data and analytics problem. Every KPI lives somewhere in your data stack -- defined in a semantic layer, calculated in a pipeline or certified in a catalog. Data teams have spent a decade governing whether those metrics are accurate. Almost no one governs whether they are right.
That is the audit to run before your next AI deployment: trace each certified metric to the business outcome it is supposed to represent. Two failures surface fast.
- The orphan metric. Average handle time is trusted and refreshed hourly, but it does not tie clearly to a business outcome.
- The unmeasured outcome. Customer lifetime value, patient recovery or closed revenue -- if they have no certified metric, then AI follows what exists rather than what matters.
Closing that gap -- connecting the data and building the lagging measures no one wanted to own -- is data work before it is AI work.
Leadership cannot be delegated
AI faithfully executes whatever task you set; leaders define what matters. Set poor objectives, and AI scales bad decisions faster than ever. Define value correctly, and it becomes one of the most powerful value engines ever built. The technology is not determining your ROI -- your metrics are. The AI ROI gap is a leadership gap hiding inside your KPIs.
Before approving the next AI initiative, ask one question: if AI perfectly improved every KPI in your organization tomorrow, would the enterprise actually gain more value?
If the honest answer is "not necessarily," then the problem was never the AI. It is the metrics. Until we measure value instead of activity, the gap will keep widening -- not because AI fell short, but because leaders asked it to win the wrong game.
Bill Schmarzo is the former CIO of Hitachi Vantara, where he was recognized for groundbreaking work in data science and automated machine learning. He has also worked at Dell EMC as CTO and as vice president of analytics at Yahoo. He has also written several books on big data.
