The AI metrics trap: Measuring speed but missing value
Rising token counts and pull request throughput look impressive, but they miss where AI creates real value: preventing bad decisions and surfacing blockers before they cost time.
An engineering leader recently walked out of a quarterly business review with a clean story: AI adoption was up 40% across the team, and token consumption was climbing. In addition, pull request throughput had jumped, indicating developers were writing and reviewing code faster. The board was satisfied.
Three weeks later, a critical feature shipped late because no one had caught a cross-functional dependency until week four of a six-week cycle. That same dependency had delayed the previous two releases.
The metrics organizations default to are real numbers: tokens consumed, dollars spent against AI tooling budgets, pull requests developers release for code changes, lines of code written with AI assistance and time to completion on individual tasks. However, they measure AI as a speed layer on top of existing work and concentrate almost entirely on the execution layer: writing code, testing it, reviewing it and writing docs. That's not an accident. Execution tasks are well scoped, inputs and outputs are clear, and correctness is independently verifiable. You run the tests and you know.
Pull request throughput went from a signal of progress to the target itself, and somewhere in that shift it stopped meaning what it used to mean.
Coding tools dominate the conversation about AI maturity because they're easy to verify. The category that's easiest to check becomes the category everyone points to as proof that the technology works. The categories that are harder to verify, and where more of the value sits, stay unmeasured because nobody has figured out how to instrument them yet.
Problem definition, solution design, cross-functional alignment and decision-making are still running at the same pace as before AI arrived. These are the stages where bad decisions get locked in, where dependencies get missed and where options get narrowed too early. None of that shows up in a token count. An organization could double its pull request throughput and still ship the wrong things, with the same upstream delays and late-stage pivots. The number is clean. The signal is weak.
This is Goodhart's law in a delivery cycle: Once a measure becomes the target, it stops tracking the thing it was meant to stand in for. Pull request throughput went from a signal of progress to the target itself, and somewhere in that shift it stopped meaning what it used to mean.
What to measure instead
The value AI creates upstream shows up as things that didn't happen -- the pivot that was avoided or the blocker that surfaced in week one instead of week four. Counting that requires different questions:
Are problem definition, alignment and decision-making getting faster, the same way execution is?
Are late-stage discoveries, bugs found in production, conflicts surfacing in week four and scope changes after alignment declining?
Are blockers tracing back to something that was knowable earlier but wasn't known? I've watched teams lose three weeks to exactly that.
A dashboard widget won't answer any of these questions. Instead, they are answered by tracking cycle time at every stage of the workflow -- execution included -- and comparing it before and after AI is introduced. That takes more time to set up than pulling a token report. It's also the only approach that tells you anything.
The questions you need to answer
None of this is trivial to instrument, and that's the point. The metrics most organizations default to are easy to pull because they sit at the end of the workflow, where outputs are visible. The value upstream is harder to attribute, and that's exactly why it must be defined before deployment, not measured after the fact.
Before approving another round of AI tooling spend, ask which of these questions the team can actually answer.
What does a well-scoped problem definition look like, and can AI stress test it before the team commits? What does early dependency discovery look like, and can AI surface it before three weeks are spent building against conflicting assumptions? It's important to answer those questions before AI touches the workflow, not after.
The next quarterly review is a reasonable place to start. Before approving another round of AI tooling spend, ask which of these questions the team can actually answer.
If the answer is none of them, that's not evidence AI isn't working. It's evidence nobody has decided what working would look like. That's a more useful finding than another quarter of climbing token counts, because it tells you exactly where to start.
Sireesha Pulipati is a data and AI engineering leader specializing in scalable data platforms, real-time data pipelines and production-grade AI systems. She is a staff data engineer at Shopify and previously worked at Google, where she led analytics and business intelligence across search and Google Cloud.
