Scaling AI puts enterprise architecture to the test
New research finds only 13% of organizations with implemented AI have scaled fully as planned, putting architecture and modernization decisions under scrutiny.
AI is producing measurable results for many organizations, but far fewer are scaling it as planned, according to a new study from management and technology consultancy BearingPoint.
BearingPoint released the study Oct. 1, based on an August survey of 1,050 C-suite executives and senior leaders. It found that 74% of organizations with implemented AI reported measurable top- or bottom-line impact, but just 13% had scaled their AI initiatives completely in line with the original business case.
The research identified complex regulatory frameworks and legacy-system integration as the two most frequently reported barriers.
An AI implementation can prove its value within a single team or workflow without proving that the enterprise around it can support that same technology at scale.
The problem isn't necessarily the AI itself. Expanding a successful implementation can expose limitations elsewhere in the technology environment, from fragmented data and legacy integrations to identity, infrastructure and workflow design.
For CIOs, that creates a different question than whether an AI implementation works: What else in the enterprise has to change before it can work at scale?
Successful AI projects are often narrowly focused, with human oversight and realistic expectations, said Lian Jye Su, chief analyst at Omdia, a division of Informa TechTarget. They tend to target a specific point of automation or optimization rather than operate across functions at enterprise scale.
Problems can emerge when organizations try to expand them. Su pointed to fragmented and siloed data, workflow-specific implementations, poor integration with legacy IT and operational technology systems, and mismatches involving governance, observability and cost controls as common fracture points.
Some of those problems aren't unique to AI.
"At the end of the day, an AI implementation is a software implementation," said Ipek Ozkaya, technical director of the Engineering Intelligent Software Systems group at Carnegie Mellon University's Software Engineering Institute.
Connecting AI capabilities to existing infrastructure can expose legacy issues, technical debt and architectural mismatches that organizations have accumulated over time, she said. Those problems may remain manageable within a contained implementation but become harder to avoid as it expands across the enterprise.
Does scaling AI require modernizing legacy systems?
Liberty Mutual has spoken publicly about using abstraction layers and APIs to make data and capabilities in core systems available to newer technologies. That approach provides significant flexibility by allowing newer technologies to use trusted business capabilities without needing to understand or change everything underneath them, according to Tony Marron, global engineering capability lead at Liberty Mutual Insurance and managing director of Liberty IT.
"That has been an important part of our modernization strategy," he said. "It creates flexibility and allows us to innovate at the experience layer while continuing to evolve the underlying technology at the right pace.
"But abstraction is not a substitute for modernization. The two go hand in hand. We still modernize underlying systems where doing so improves resilience, security, data accessibility, speed of change or business capability."
Strong abstraction can also give enterprises more flexibility to modernize underlying systems based on business and technology needs rather than rebuilding them simply to accommodate a new AI capability, Marron said.
Abstraction is not a substitute for modernization. The two go hand in hand.
Tony Marronglobal engineering capability lead, Liberty Mutual Insurance
But deciding whether to modernize an underlying system or continue building around it depends on what that system will be asked to support.
There is no single threshold for when an older core system needs to be modernized rather than accessed through APIs, wrappers or abstraction layers, Ozkaya said.
CIOs need to consider how frequently a legacy system exposes bugs or security risks, how difficult it has become to maintain and operate, whether it can expose the capabilities AI systems need and whether its data dependencies can be effectively managed, Ozkaya said. They also have to weigh the cost of modernization against the cost and risk of continuing to operate the system as it is.
The technical requirements of the AI implementation can help determine where that tipping point lies. APIs and abstraction can work well when AI systems are primarily reading from older systems or providing decision support, Su said.
The calculation changes when AI requires subsecond response times, the ability to write transactions back into core systems or high-volume data transfers. Those requirements, along with rising integration and maintenance costs, can make modernization of the underlying application or infrastructure necessary, Su said.
The insurer lets consumers obtain an auto insurance quote through a ChatGPT app. Key quoting capabilities had already been modernized and exposed through APIs, Marron said, allowing the conversational experience to connect with trusted business capabilities already running in production.
As Liberty Mutual expands approaches like that, the work increasingly shifts toward building shared capabilities around the AI itself, including identity, security, evaluation, observability, data access and common integration patterns, Marron said.
The goal is to avoid solving the same problems repeatedly for each implementation and turn what the company learns from one implementation into reusable capabilities for the next.
Those dependencies become more consequential as AI systems grow more complex. Poorly managed dependencies at the system, software and AI component levels can surface at runtime as "cascade failures, unexpected latency spikes and silent correctness bugs," Ozkaya said.
Assessing AI scalability before launch
Some of those scaling problems could be identified earlier. Fewer than one-third of organizations formally assess scalability before launching an AI initiative, according to BearingPoint's research.
For CIOs, that means evaluating more than whether the AI can perform the intended task. Su said organizations should assess whether the existing architecture has the right economics and data support, how easily AI can be integrated and whether identity management, policies and guardrails are in place. They should also determine whether the architecture can support changes to business logic and workflows that come with the AI implementation.
Liberty Mutual takes a similar approach by looking at the systems an AI capability will need to interact with, the data and context it requires and the level of access and authority it should have, Marron said. The company also considers where human oversight is required, how the capability will be evaluated and monitored and which existing enterprise capabilities can be reused.
Not every successful experiment should become a production capability. But when an AI implementation shows value, CIOs need to determine whether the surrounding technology environment is ready to support the same capability at a much larger scale.
That can make scalability an architectural question from the beginning rather than a modernization problem discovered after an AI implementation has already succeeded.
For a closer look at how one enterprise is moving from AI pilots toward scaled deployment, watch this episode of IT Ops Query featuring Zach Womack, CTO of financial services firm SEI. He discusses the architecture, workflow redesign and governance required to expand agentic AI across the business.
Liz Hughes is an award-winning editor and writer covering AI and emerging technology and the former editor of AI Business and IoT World Today.