SMBs face AI governance reckoning

The push to adopt AI throughout the enterprise marches on. But despite the amount of money pouring into AI projects, many organizations are failing to see any returns on the investment.

Even more troubling for organizations, mistakes that occur from AI failures can result in lost business and damaged brand reputation. This might be disappointing for large enterprises, but it can be potentially serious – or even catastrophic – for small-to-medium sized companies (SMBs).

These companies are eager to realize the productivity gains that AI can deliver but lack the resources to deal with AI initiative failures, according to Arun Sahu, head of AI data and applied intelligence at Alliant, a Houston-based provider of digital transformation and managed services.

In this Q&A, Sahu explains that one of the main problems that SMBs must deal with is "AI psychosis," where AI produces results that are questionable at best and completely false at worst. SMBs generally lack the expertise and resources that large enterprises should provide the AI governance that can mitigate the costs of AI failures.

Editor's note: The following transcript was edited for length and clarity.

What are the characteristics of AI psychosis in the enterprise?

Arun Sahu: One characteristic of AI psychosis is that AI is very convincing. It feels like it can do everything, and we're relying on it a lot now. If Claude, ChatGPT or any kind of AI is giving an answer to organization, you are assuming that it's always right and it can find things faster. There's an expectation that it will be able to solve all your problems. This is the AI psychosis that I believe we are experiencing today. We think that an email or a document written by AI is of top-notch quality, and so on.

Arun Sahu

Why is this a bigger issue for SMBs than it might be for a large enterprise?

Sahu: When it comes to SMBs, [AI psychosis] has a bigger impact than enterprise level because the buffer is less. Enterprises have manpower, they have people who can proactively review if something goes wrong. And if things do go wrong, they have the buffer to accommodate it. When it comes to SMBs, this buffer is almost none. Let's say you have an organization of 30 or 40 people who are trying to automate everything through AI. They see all these messages that today what AI can do, for example, it has automated the entire finance function. So, they adopt it, but they don't have the buffer to keep monitoring the edge cases where it has gone wrong, because everything the AI produces is very convincing. If something goes wrong and ends up in a client's inbox, they'll have a bad relationship. Losing clients to bad experiences has much more impact on an SMB than an enterprise.

So,  SMBs don’t have the resources to put governance on AI like enterprises do?

Sahu: Yes. Typically, I call it the low safety net reality, where small businesses don't have IT risk teams to mitigate AI edge cases or AI failures. So, the top management or the people who are trying to leverage it, become AI managers. Ultimately, they have to monitor the entire thing. They are forced to do it.

How is AI coming into SMBs?

Sahu: From what I have experienced, SMBs usually do not have a CIO, or maybe they're using a partial CIO. So, it's typically the people around who are looking at improving productivity, like the unit or practice heads. The easy way today is to just go ahead and try out some tools without any safety net, and then they see that the outputs are very convincing. So, without doing a deep dive on it, they say "why don't we just purchase it, because it's doing this for me and it can do everything. I've spoken to some of our colleagues and I believe that it can do it."

But what happens is that it fails the last mile of the 80-20 rule. It's able to do that 80%, but that's not as impactful. The 20% -- the last mile -- is always a problem. You hit that wall and then you spend most of the time trying to fix or handle those issues. This is called the productivity tax, which is a quite common outcome that I see from SMBs. They don't have the expertise in-house plus the budget and the time to do all the deep analysis before anything can be rolled out or adopted.

What are some specific negative effects of AI psychosis on SMBs?

Sahu: For one example, we were working for an oil mining company that has a lot of equipment.  Instead of going to the warehouses [which are spread out geographically], we could fly a drone to take pictures of the vehicles and then create an inventory using computer vision and agents. Can that happen? Yes. But when there's an edge case, like if it was cloudy or there is a dust over the labels, the agents might start hallucinating. And as soon as they hallucinate, it screws up the entire inventory. A large enterprise can manage -- but imagine if this entire thing is done at a smaller company where inventory is important. This can be very dangerous when it comes to finance or inventory management. Imagine if this is a critical thing like bringing oxygen cylinders to hospitals. That kind of thing is a potential side-effect of AI psychosis when people just trust and let their system be run by the AI agents.

So, one big problem is that people are simply too trusting of what AI agents tell them?

Sahu: Yes. AI is trained to be convincing. My theory is that we shouldn't grant AI agents any unaudited read or write access to critical information. That's where we need to be quite careful.

How should SMBs deal with this if they don't have the budgets to put governance around AI?

Sahu: First, for an SMB it's very important to not automate the chaos. Sequencing your AI adoption is important. It's important for an SMB to bring AI to their data, but they must understand their data first. Work on the data structure and start automating single high-volume tasks. Go for simple, single tasks for some time, then start building those automations and monitor them. You can eventually start to scale them because there are fewer error rates when it's a simple and single task.

The problem that I see today is with all the hype -- although I don't believe that there's an AI bubble -- but with people talking about being in an AI adoption race. I've seen organizations claiming that they have adopted AI, and they are saving so much in productivity. This creates a good story, but the reality is different. It creates a competitiveness problem, and the SMBs feel like they can just buy a tool that can do all of this. So, they start chasing productivity, which is a problem when you don't have the right resources and right capacity behind you. You start chasing a platform AI solution to improve productivity. It's a bigger challenge.

Because if you have a complex process like order-to-cash, for example, one small error in the beginning could be a huge issue as it scales.

Sahu: Yeah, absolutely. Today people talk about the human-in-the-loop, but the reality is it's not really a human-in-the-loop, it's usually a human who is just sitting out there doing the rubber-stamping. For example, there's a story where one of the big consulting firms sent out a report in Australia, but they were penalized because it was AI-generated and nobody reviewed it. SMBs need that real human-in-the-loop who is active and is putting the right information in for the AI agent.

SMBs may not be able to have a full-time CIO or a team that can manage AI. Should they always have someone in the organization be the person responsible for understanding what's going on with AI and be the governing entity over it?

Sahu: Yes, I would strongly recommend that because today we are dealing with a probabilistic system. You need some human expertise who would be the decision-maker or is knowledgeable enough to evaluate a decision.

What should SMBs be thinking about with the evolution of AI for the foreseeable future?

Sahu: I don't see this to be an AI bubble that will burst any day, and it's not going away. So, we need to be a little more conscious. We cannot just assume that AI can do end-to-end transformation or do anything like it's on autopilot. You should be very conscious on how you plan your roadmap and how you enable the workforce behind it to take AI's decisions that ensures it doesn't become a liability. You want it to be an asset that you can bank on, and you're not paying a productivity tax on it. 

Jim O'Donnell is a news director for TechTarget, where he covers IT strategy and enterprise ESG.