CIOs face an AI expertise paradox as automation reshapes experience
AI can increase demand for experienced human judgment while automating the work that helps create it. CIOs need to plan for the expertise pipeline AI still depends on.
AI is fast. It can learn, produce and process information at a scale no employee -- or group of employees -- can match. But AI also generates a lot of noise.
Sorting through that noise requires topic-specific expertise and judgment. The paradox is that AI automation can narrow some of the traditional avenues through which entry-level and less-experienced employees develop that same expertise.
Anthropic's Project Glasswing shows how that tension can play out in practice. Its AI-assisted vulnerability work generated 26,153 findings. Only 202, or 0.8%, were ultimately marked fixed. Claude classified 91.5% of one group of findings as high or critical, while human maintainers put only 51.3% in those categories.
The issue wasn't whether AI could find vulnerabilities. Clearly it could.
Deciding which findings actually mattered required human judgment. That meant experienced people who understood the software, the context and the difference between something that looked alarming and something that actually warranted action.
Those people have to come from somewhere. And as AI takes on more of the work that once helped employees build experience, enterprises may need new ways to develop them.
What experienced people do have is a particular kind of judgment in their areas of expertise, built up over years of actually doing the work.
AI can know more without knowing better
The veteran security analyst, developer, finance analyst or IT operator doesn't know more facts than an AI system.
How could they?
No matter how smart a person is, there are biological limits to how much knowledge people can store and how quickly they can access what they know, let alone everything available through a computer screen.
What experienced people do have is a particular kind of judgment in their areas of expertise, built up over years of actually doing the work.
They know when an answer looks technically right but feels wrong. They recognize an exception because they've seen something like it before. They understand which rule can bend and which absolutely cannot. They know when information is missing, can anticipate downstream consequences and sometimes know when not to act.
That kind of judgment is important to their daily work. It becomes critical when AI is involved.
You can't treat AI as inherently trustworthy simply because its answer looks authoritative or the task appears to have been completed successfully.
AI can hallucinate. It can miss context. It can mishandle an exception. It can reach a technically plausible conclusion without understanding all the consequences that follow.
And some of the most important guardrails are experienced people.
Cybersecurity leaders are already confronting that problem. Richard Horne, CEO of the U.K. National Cyber Security Centre, said his organization intentionally lets analysts continue doing some work that could be fully automated because those analysts need experience to develop the judgment required to oversee AI effectively.
That matters because AI doesn't just require expertise. It can also change the way that expertise gets created.
AI takes grunt work and experience with it
AI taking over entry-level grunt work is a good thing. I don't think many people would argue that employees should keep doing inefficient manual work simply because previous generations had to do it.
The issue is the real learning that happens on the job.
A security analyst learns by working through false positives, ambiguous signals and incidents that don't behave the way the textbook says they should. A developer learns by debugging code and figuring out why something that looked right didn't work. A finance analyst learns by reconciling numbers, investigating exceptions and discovering why an answer that technically adds up still doesn't make business sense.
That is how knowledge gradually becomes judgment. And that creates a problem for enterprises.
AI may depend more heavily on experienced human judgment while changing some of the mechanisms through which businesses develop experienced people.
AI can perform entry-level work and routine workflows far faster and often more cheaply than an entry-level employee or even a group of them. But if AI removes some of those tasks, how are today's inexperienced employees going to become tomorrow's experienced ones?
And where are businesses going to find the experienced people they need to provide context, evaluate AI output and make the final decisions?
Recent TechTarget reporting on disappearing career-ladder rungs raises exactly that concern. Teikametrics CFO Brian Beaupre spent years early in his career building financial models -- work AI can now perform much more quickly. His concern isn't that younger workers need to keep building models by hand for tradition's sake. It's that doing that work helped him learn when the answer was wrong.
That's the collision. AI may depend more heavily on experienced human judgment while changing some of the mechanisms through which businesses develop experienced people.
Don't save the grunt work -- replace the learning
There is an obvious response to all of this.
Technology has always eliminated old work. People learn in new ways. Companies can put new training and development programs in place.
Yes -- up to a point.
Nobody needs to preserve obsolete work merely to build character. But eliminating the work doesn't automatically replace the learning that happened while people were doing it.
The task is not the lesson
Entry-level work has never been valuable simply because somebody had to do it. Its developmental value comes from exposure to mistakes, exceptions, ambiguity and consequences. AI can take over the task without automatically replacing those experiences. The challenge is not deciding which old work should survive but identifying what future experts still need to learn from doing it -- and finding another way for them to learn it.
Career development isn't new, and it has never been a one-size-fits-all process. AI simply adds a new factor: Employees may no longer gain experience from processes and tasks that increasingly belong to AI.
You can put plans in place to mitigate that loss. But first somebody has to recognize what was lost. Then somebody has to design the replacement.
Some organizations are already experimenting.
As mentioned earlier, the U.K. National Cyber Security Centre is deliberately preserving selected hands-on work so analysts can develop judgment. At Teikametrics, experienced workers are being paired with newer employees, with senior employees explaining how they reached decisions while younger workers help them use AI tools.
Those are two answers. Neither is the right one -- at least not yet. This is still too new for that.
Automation decisions are also development decisions
For CIOs, the question shouldn't simply be whether AI can do work faster or cheaper. The bigger question is what should change in the way enterprises evaluate automation.
Two things complicate that evaluation. First, AI work can look right and still be wrong.
Second, experts -- along with orchestration, contextual awareness, governance and other safeguards -- are still required to evaluate both the automation process and its results.
That expertise often grows from the ranks of less-experienced employees as they acquire not just knowledge, but the ability to judge and comprehend the issues within their domain.
The business-function leader should know what skills and judgment employees develop by doing that work. The CIO knows what the automation is changing. They need to work together and ask a few questions while the automation is being designed and put into production, not after the development path has disappeared.
What did employees learn by doing this work?
What developmental capability disappears when AI takes it over?
And if AI now does the work, where will that learning happen instead?
The answer isn't to stop automating. It's to understand what else is being automated away with the task.
After all, enterprises still need experienced people to make sure the AI is doing the work right. And today's inexperienced employees are where many of those experts are supposed to come from.
James Alan Miller is a veteran technology editor and writer and Lead Editor for CIO News at Informa TechTarget. He directs coverage of enterprise technology strategy, AI, software, data, infrastructure and the decisions shaping how CIOs manage increasingly complex IT environments.
