How CIOs move from AI experimentation to operationalization

Businesses are now shifting from AI experimentation to operationalizing AI at scale.

That was the topic of discussion during a panel at the annual MIT Sloan CIO Symposium in Cambridge, Mass., on May 19.

But what does it take to make scaling possible? It starts with access and tracking how people in a business are using it, said Mojgan Lefebvre, executive vice president and chief technology and operations officer at Travelers, during the panel. It then includes executives prioritizing which use cases need to be scaled and the discipline to say something doesn't work or can't scale.

Brook Colangelo, CIO at Waters Corp., a life sciences and diagnostics company and another panelist, concurred, saying rigor and scrutiny are crucial.

But humans are still part of the equation, and AI might not always deliver value.

"Not every technology solution is made for every problem," said Lefebvre.

Businesses also need to think about economics before jumping into scale AI adoption, or they might find themselves in trouble, Lefebvre said.

"If you're looking at head count, you also need to look at your AI costs," said Amy Farrow, CIO at Celigo, an intelligent automation platform and also a panelist. "There are a lot of things that need to work in concert. It's a company transformation, not a technology transformation."

We sat down with Farrow after the session to ask her some additional questions.

Editor's note: The following transcript was edited for length and clarity.

What was the pivotal moment or signal that told you it was time to shift from AI experimentation to full-scale operationalization?

Amy Farrow: I think that's a good question, and I'll answer it a little bit hypothetically, because I've been at Celigo for less than two months. Here are some of the things we've done and some things I've done in my past roles. I think what it really looks like is you have one proven enterprise value -- like that there's a business outcome tied to this initiative. That there's an owner for it. That you have the right architecture and governance and a minimum amount of controls in place to feel confident that data accessibility is managed, observability is there and reversibility is there. And that you have a plan for when things go wrong.

Can you walk me through your AI governance framework?

Farrow: We are actually in the process of standing up an AI governance council. It's one of the things I'm standing up as having joined the company. But we do have it defined in terms of security controls, in terms of access controls, in terms of API and MCP controls. But, AI governance means a lot of things to a lot of people. So, the reason I want to stand up [establish] a council is, I think we need to look at it holistically -- from a cost management, use case and risk management perspective. Then look at it from a scalability perspective. So, some of those components are in place today, but they're not looked at holistically across the organization.

What are the most critical technical infrastructure components you've had to build or upgrade to support enterprise-wide AI operations?

Farrow: I think the most important components for AI enablement across the enterprise are investing in your data foundation, building data products and exposing them to both human consumption and agentic consumption. I think our own product at Celigo, an integration layer capability, where you have the right controls over the right access -- role-based access control -- you are also able to segment what data an agent can have access to. And you're also able to monitor and implement guardrails and leverage that across multiple systems, not just within one. And our security -- like stack and infrastructure -- is also an important part of having confidence in deploying enterprise AI.

How have you redesigned workflows and business processes to incorporate AI capabilities effectively?

Farrow: I think we're on a journey. During the panel, I talked about level one, level two and level three. Level three is aspirational -- where we want to get to -- which is AI first, really redesigned and reimagined. I would say we are more at level two today, in the sense of creating agentic insertions into workflows, potentially with a human in the loop, autonomous where possible, where we deem the risk to be low, and where we find we can contain the nondeterministic nature of AI. But we are continuing to pursue that level three journey. One of the things that I'm working on is looking at our operating model and really co-building with different businesses, so we can really realize the value of enterprise AI -- understanding the right outcomes and building with the context of how this functioning organization should run. So, that's, I think, for me, going to be the unlock for more level three.

What's the biggest mistake you made during the transition from experimentation to operationalization, and what did you learn?

Farrow: I don't know if it's the biggest mistake, but I think, like any technology deployment and change, the process change that goes along with it -- the enablement, the change management, the behavior change -- is a really important part of making it successful. And the areas where we've seen more success, we had that in place. And the areas where maybe we deployed a solution, but we didn't quite see the value, we weren't as strong in those areas.

How do you measure the success of operationalized AI beyond traditional IT metrics?

Farrow: It's hard to do it in the abstract. I don't think there's a simple metric here. I think for each function in each area that we're trying to optimize -- which use case we're trying to optimize -- we should be able to develop those metrics. But if I were to abstract it into a more general concept, I think of it as business-process velocity. So, if you're in finance, that could be closing the books. If you're in go-to-market sales, that could be closing a deal or renewal. I think the more velocity we get in terms of the quality and delivering that same outcome faster, the more meaningful impact to the business. We also think about scale. So, can we scale revenue per employee -- that really shows that we're leveraging capabilities in the right place. That's giving us a better kind of scale curve -- whether that's just to grow the business further with the staff that we have, or something else.

Sarah Amsler is a senior managing editor for the IT Strategy team at TechTarget.