CAMBRIDGE, Mass. -- For some businesses, investing in generative AI might feel like taking a leap of faith.
Even so, it's a leap many organizations are taking. In Deloitte's 2024 generative AI report, 78% of respondents said they expected to ramp up overall AI spending in the next fiscal year, with generative AI accounting for an increased share of that budget.
The sentiment was loud and clear this week at EmTech AI, hosted by MIT Technology Review: Generative AI investment can lead to business value. However, leaders need to put the right structures in place -- like change management, employee training and clear success metrics -- to ensure that those investments succeed, rather than wasting time and resources.
A new approach to ROI
While conventional ROI metrics focus on financial investment vs. gain, generative AI's initial value overwhelmingly comes from improved efficiency.
These tools will make existing employees more efficient at their jobs. What we expect is an increase in productivity.
Andrew LoDirector of the Laboratory for Financial Engineering, MIT Sloan School of Management
Andrew Lo is a professor of finance and director of the Laboratory for Financial Engineering at the MIT Sloan School of Management, who researches generative AI and financial technology. "These tools will make existing employees more efficient at their jobs," Lo said in an interview with Informa TechTarget. "What we expect is an increase in productivity."
For example, the financial industry is increasingly using generative AI for investment banking and fundamental analysis, Lo said. Several banks are experimenting to see how generative AI can improve workflow efficiency.
Thanks to their natural language processing, coding and automation capabilities, large language models (LLMs) can dramatically increase how much data a human analyst can evaluate.
"Instead of focusing on maybe 10 or 20 companies as part of their purview, one human analyst can now look at 100 or 200 companies," Lo said.
Productivity gains are also increasingly coming from AI agent use, with some companies starting to adopt an "agent-first" approach, said Asha Sharma, corporate vice president and head of Microsoft's AI Platform, in an interview with Informa TechTarget. An agent-first mentality means considering whether an agent might be eligible for use when creating any new project.
Understanding how to measure generative AI's ROI
In the session "The Real Impact of AI on Your Organization," Jim Rowan, head of AI at Deloitte,discussed how he advises clients to identify generative AI ROI.
"We're really looking at the value creation from use cases," he said during the session. That value often doesn't come in the form of immediate cost savings -- instead, it's value from time saved or quality added.
"When you accelerate your [R&D] pipeline, it's not just about cost," said Gabriele Ricci, chief data and technology officer at pharmaceutical company Takeda, in the same session. Instead, he said, Takeda puts equal focus on how it can bring better-quality medicine to patients faster.
When it comes to generative AI initiatives, ROI can be a bit of a waiting game, Lo said. In the short term, businesses might see neutral or even negative ROI due to the initial costs associated with building infrastructure and employee expertise.
But over time, Lo said, businesses will start to see returns in the form of productivity gains. In the long run, that means cost savings from replacing divisions and cultivating highly trained employees who can use AI in significant and creative ways.
"It will take time to be able to see the fruits of these labors," he said.
Andrew Lo discusses the main stages of generative AI ROI, including short-, medium- and long-term returns, during his session, 'Monetizing AI for Business Growth' at MIT EmTech AI this week.
Embracing change management
Effective change management is a crucial element of generative AI ROI. Generative AI increases efficiency only if businesses know how to repurpose the time it saves.
"[Employees] don't know what to do with that extra time," said Kevin Bolen, head of AI transformation at KPMG, during the session "AI Strategies from the Front Lines."
Without proper guidelines for repurposing time, workers' responsibilities can be unclear, potentially leading to anxiety and diminished efficiency. Bolen suggested that businesses explicitly outline how workers should repurpose their time to enhance their productivity and contribute to company goals.
Change management also includes preparing for the future. Organizations that keep employees in the dark about generative AI initiatives are likely to face resistance.
"Culture is one of the most difficult things to define, manage and change," Lo said. Senior leadership is responsible for communicating the business' vision for generative AI, developing clear objectives and keeping employees informed on how the technology is introduced into workflows.
Commitment to upskilling is another critical element, Lo said. For employees who might experience more substantial job changes due to generative AI, companies must have a transition process that includes further training to keep them involved when their job focus changes.
"[Businesses must] adopt a change management philosophy … that is both compassionate for the people that are going through this transformation, but also realistic," Bolen said.
Consider technology limitations
With any generative AI investment, businesses must consider how to best account for the technology's limitations. For example, LLMs hallucinate -- a term used to refer to false or misleading information produced by generative AI models. That's a big issue for businesses, especially in high-risk industries.
Likewise, LLMs' inability to deal with numerical information is a significant limitation in finance, Lo said. Teams can combat this by ensuring analysts check model output vigorously and using LLMs for summarization and analysis, not to make precise mathematical calculations.
Turning investment into value
Planning an AI initiative that results in substantial ROI can seem daunting for many businesses.
Organizations can get started by creating sandbox initiatives and other test environments to explore generative AI applications, Rowan said. "We have to give our employees the opportunity to express their curiosity," he said.
The most effective starting point is often the repeatable actions that employees take on a day-to-day basis, Sharma said during the session "Turning AI into Measurable Business Value." These are the areas where generative AI is most likely to reduce the energy, time or resources needed to complete a task.
These initial investments don't need to be huge, either, Sharma added. Businesses can start with safe, small proofs of concepts and scale them over time.
"The No. 1 piece of advice I have is [to] constrain your budget," she said during the session. "Without constraints, it doesn't force you to innovate and truly push the technology to its limit."
In her session, 'Turning AI into Measurable Business Value,' Asha Sharma, left, discusses generative AI use cases and investment strategies with MIT Technology Review senior editor Will Douglas Heaven.
To turn their initial investment into business value, businesses must actively measure metrics, Lo said. Senior management needs to define appropriate metrics for calculating ROI and use those factors to continuously measure the generative AI initiative's effectiveness.
During his session, Rowan encouraged business leaders to get creative with the ROI metrics they choose to measure. That could mean a mindset shift toward calculating efficiency and productivity gains -- for example, measuring output timelines and qualitative benefits.
Sharma said in an interview with Informa TechTarget that metrics such as margin and cost per unit can also be used to calculate efficiency. "My advice and encouragement is to look at the unit economics, because that's what will allow you to scale the investment," she said.
Ultimately, choosing which metrics are appropriate for calculating generative AI ROI will depend on a specific business and its goals.
"There are many different ways of measuring impact," Lo said. "That needs to be decided upfront, because you can't manage what you don't measure."
Olivia Wisbey is the associate site editor for SearchEnterpriseAI. Wisbey graduated from Colgate University with Bachelor of Arts degrees in English literature and political science and has experience covering AI, machine learning and software quality topics.
