Why pragmatic approaches to AI can minimize pilot failure

According to the Massachusetts Institute of Technology, a staggering 95% of organizations are failing to see a return on their GenAI investments. These failures increasingly point the finger at workflows and architectures, not AI technology itself.

In this Q&A with TechTarget, Chetan Saundankar -- chief executive officer, Plant360.ai, and founder, Coditation Systems -- explores why this is, and shares how IT leaders can increase the likelihood of their AI pilots graduating from the sandbox into production.

Editor's note: This Q&A has been edited for clarity and conciseness.

Why do so many AI pilots fail?

Chetan Saundankar: I run two companies, so I get to see and work with a lot of customers. I see a lot of failures, but also a lot of successes. My observation is that the AI technology itself is rarely a challenge, particularly post-2024 when models are more sophisticated. I find that it is the workflows -- by which I mean data automation and human workflows -- that are broken or wrongly optimized for human-led operations rather than AI-led ones.

Plant360.ai is an AI platform for oil and gas, energy and utility organizations. We find that even if a company automates one part of a workflow, the end-to-end process is not optimized, and so it still takes the same amount of time. The project sponsors and business execs then ask, "Did AI actually solve my problem here?" But the problem was not technology; it was the process. That's why businesses need to adapt workflows and processes for automation.

Where should organizations start when identifying where they need to adapt workflows and processes for AI?

Saundankar: In enterprises, an exec often tells his or her team, "We need to do an AI implementation this year -- this is something I've promised.” Teams then think, "Here is a tool I have in my hand, and now I'll start searching for a problem for it."  This is how, unfortunately, AI projects become hobby projects. You need to start by identifying a problem or friction; otherwise, you'll jump into implementing AI. There have been cases where the problem did not even exist to the extent that it warranted automation.

AI pilots also fail because teams are not properly set up with the right business and operational metrics. Teams will never know from pilot or proof-of-concept data whether it succeeded in achieving what it was supposed to do if it is evaluated on a hunch rather than data.

AI models have become extremely savvy and smart since 2024. But with so many of the MIT survey respondents saying their pilots did not succeed, the gap is clearly in the last mile -- how you adopt, how you position and what problems you're choosing to solve.

When customers realize it's the process that's broken, not the AI, what are the early warning signs they miss that you see?

Saundankar: We insist on a "discovery phase" at Coditation and Plant360.ai. Even before taking up the first pilot, and even if that pilot is just a couple of months, we do about a month of discovery. These discovery phases are where our consultants and analysts focus on the processes and flag, "This seems broken, we're not sure this can work.” The reason is to understand and weed out the problems. If you have multiple variables with question marks post-pilot, you can't do a proper root cause analysis, and you may end up drawing the wrong conclusions.

Then it's the customer’s call whether to go ahead and prove the technology anyway, address the process as its own program, or do both at the same time. In enterprises, the challenge is that these efforts are often disjointed and siloed. One effort is looking at the process, and a separate effort is looking at AI adoption. If they don't merge, there's always a risk of misalignment.

What advice would you give for merging and breaking down these silos?

Saundankar: The successes we've seen come down to having a highly motivated leader and sponsor who understands that siloed programs will cause more friction. Those who say, "I'm working with multiple partners, but I want to bring them into the same room: there's got to be collaboration. " Aligning this way is still a pipe dream in many large enterprises, but there are enterprises making a very conscious effort to do that, and it pays off really well.

I'd say almost 40% to 50% of CEOs and execs realize the problem might be somewhere in the data, the processes, or the people.

With so many pilots failing, how are CEOs and execs responding?

Saundankar: The AI era is nascent -- we're probably in the third or fourth year. Some companies are learning quickly and responding to failed pilots or rollouts, and others are taking a step back, saying, "We want to wait this out and think this through."

I'd say almost 40% to 50% of CEOs and execs realize the problem might be somewhere in the data, the processes, or the people. Because even if you're building these AI agents out, at the end of the day, it is people who are using them.

It’s also about ensuring the execs and sponsors understand where the gap is. CEOs, CXOs, or sponsors are not dismissing AI, saying "the technology doesn't work." By and large, they understand that there must be something they're doing wrong rather than blaming it on the tech itself.

Do you have an example of a successful pilot you've seen?

Saundankar: Right now, AI in general is geared toward productivity and efficiency. Plant360.ai has one of the largest nuclear owner-operators in North America as a customer. They have drawings dating back to the 1970s -- literally hand-drawn drawings. There's a nuclear project coming up that's supposed to use the same design. One of the challenges is that digital-first -- not even AI-first, digital-first -- engineering requires this data to be in a "smart" format, so that a lot of downstream tasks can be automated.

This customer and sponsor have been extremely motivated. During the pilot, we saw an efficiency improvement of about 52%. They're now going into production, and digitizing all their plant data. There were bumps in the road -- such as their team needing to get used to the product -- and questions about whether it would scale, but the sponsor and exec were convinced this was the right way to go. Now we're in production with them, and efficiency is only improving, not declining.

This is one of the cases I cite often: conviction, a leap of faith, and a rightly designed pilot were extremely important to taking that into production and scaling the system.

How about one that wasn’t successful?

Saundankar: I wouldn't call this a failure, as such, but rather an example of what happens when you don't design a pilot well. When designing pilots, you want to track the operational metrics, not the most important business metric overall to your organization.

For example, if your network operations require 10 agents to be developed, in the pilot you're only going to develop a couple to deploy and see what they bring. Each agent brings an incremental efficiency improvement of a couple percent, so unless you have all 10 agents in production working together, you're not going to hit the 25% or 35% mark you wanted overall. If the pilot is designed to track your ultimate business metric of 25% or 35%, you will end up seeing only 4% or so improvement, and execs will say, "This isn't good enough."

These are some of the important factors in making pilots and adoption successful and meaningful.

I think we'll figure it out, move on to the newest challenges, and there'll be a net gain of jobs rather than a loss.

There's a lot of anxiety around AI taking jobs. Are you finding that customers want to use AI to free up resources to use elsewhere, or is it about cost reduction and layoffs?

Saundankar: The jury is still out. Klarna was all over the news for various reasons, claiming they'd replace all their SaaS to partially walking that back. Klarna's example is a great one where they ended up substantially or fully reducing their customer support and customer service teams, and from what I know, they had to hire them back because the CEO said, "We went too far." Their story is a good reminder that AI hasn't become autonomous yet.

One of the approaches we take, for both Plant360.ai and Coditation, is "pragmatic AI." This is because vendors go around saying, "You're going to cut your costs by 90%, 95%; this is fully autonomous." That's not the case. The notion that AI can fully replace, at least as of now, isn't zero or one.

I think there will be losses, particularly in software engineering. We see a 50% improvement, meaning that the same amount of work can be done with fewer people. Having said that, my personal view is that, in the medium- to long-term, all technological revolutions have brought more work. It has never reduced the work -- the Industrial Revolution, dot-com, cloud migration, all of these. AI feels different, for sure, but I don't think there will be job losses in the medium- to long-term. I think we'll figure it out, move on to the newest challenges, and there'll be a net gain of jobs rather than a loss.

Jobs will look very different, just like farming after the Industrial Revolution. I believe there'll be a net addition, not a reduction.

Do you think that long term, the future looks brighter?

Saundankar: The period when disruption happens tends to be a period of upheaval, and we're currently going through that. Part of the job losses, in my opinion, also has to do with the post-COVID phenomenon, where digital companies ended up hiring a lot more because of an unnatural spike, and some of these corrections are down to that. So, it's a combination of both.

That said, in the short-term, we cannot look away from the impact. The June report showing the most job cuts in U.S. tech in two years shows that the impact is real. However, my guess is that this will be short-lived.

Harriet Jamieson is a senior manager of custom content and writer for the IT Strategy team at TechTarget.