NEW YORK – Enterprises continue to pour money into AI pilot projects, yet problems persist that prevent them from putting agents into production.
Gartner estimates that worldwide spending on AI will total $2.5 trillion this year and rise to $3.3 trillion in 2027 after totaling $1.8 trillion in 2025. Meanwhile, most AI initiatives still deliver little return on investment.
Governance remains perhaps the most prominent barrier to more successful AI development, according to Cloudera chief technology officer Sergio Gago. Fragmented data estates, compliance concerns and a scarcity of GPUs are other persistent problems, but nothing trumps the struggles enterprises are having as they attempt to govern AI.
"Having an extremely robust governance system on your orchestration platform that controls all the data, regardless of where it is, is the most important barrier that companies need to overcome these days," Gago told TechTarget during a wide-ranging interview at Evolve, Cloudera's annual user conference.
Gago noted that Cloudera's newest capabilities -- Cloudera Blueprints to provide customers with AI development accelerators and Anywhere Cloud to enable users to build and scale data AI applications across any environment -- are designed to address the many problems enterprises face when building AI tools.
In addition, given the emphasis Cloudera has placed on the concept of evolving, he discussed the evolution of data management and what it means to adapt to the current era of AI, how Cloudera attempts to distinguish itself from competitors such as Databricks and Snowflake, and what he predicts will be data and AI trends a year from now.
Editor's note: This Q&A has been edited for clarity and conciseness.
During Evolve, Cloudera introduced Blueprints to simplify AI development and highlighted the upcoming launch of Anywhere Cloud to enable hybrid and on premises AI deployments -- does customer feedback provide the impetus for product development, or is it something else?
Gago: It's a combination of things.
Traditionally, customers have come to Cloudera, said that they have 50 terabytes of data, and asked if Cloudera can build a bigger platform that they can deploy in their data center or on the cloud. That's our business, and we say yes. But customers are very technically oriented. Where the world is going today with AI, it's more use-case oriented.
As an example, customers have data coming from sensors around the world, and they want to find patterns at the edge so insights can be sent back to a centralized system where they can do forecasting for maintenance. So, now, we're talking less about databases and more about use cases that solve real business problems. The third lever after customer feedback and addressing use cases is … [ensuring that humans] extract value from AI.
In a way, [the overall impetus] is the faster horses theory. We didn't want to just create faster horses with a new version of [the Impala query engine for Hadoop]. We want to create the car when no one is asking for it.
With customer feedback part of Cloudera's motivation for product development, what are you hearing from users as they try to build AI agents that can be trusted in production -- what are the biggest problems they face as they try to evolve?
Gago: There are many challenges, and different regions of the world prioritize them differently.
Procurement is one challenge. Hardware is limited, and the supply chain is stagnated, so access to GPUs is definitely a challenge. Most companies have a very fragmented data estate. Many companies have two different universes, the public cloud and the private cloud, and the two usually don't talk to each other, so they are trying to break siloes and make things work together. In Europe, there's the concept of sovereignty and private AI, making sure that we are the owners of our own destiny, that we do not depend exclusively on two or three companies, and that the data is properly kept within a controlled environment.
This year, the return-on-investment conversation has picked up. Companies are creating beautiful use cases that have a 'wow' factor, but where is the money?
Sergio Gago
Chief technology officer, Cloudera
And this year, the return-on-investment conversation has picked up. Companies are creating beautiful use cases that have a 'wow' factor, but where is the money?
Are there any other common themes you're hearing from customers?
Gago: The last one, but not the least in terms of importance because this one is global, is the governance of the data.
In the lab, you create a super nice agent that promises 50% operational gains, or whatever. But then you start looking into the governance of the data -- how was it built? When you test the system, if Customer A using the agent can see product information from Customer B, you realize that it's not fixable because [the problem is part of the] prototype.
That's where we see many AI use cases failing -- compliance, legal, cybersecurity. The governance of the data is a fundamental consideration that experimentation teams leave until the end, and then the use cases fail. You have to consider the governance of the data as a core component, a core use case of what you are building. But the difficulty with agents is that there is a secondary level.
There has to be control of the type of data they can view and how the data is provided.
What is the significance of the word 'evolve' to Cloudera -- why make that the name of your conference?
Sergio Gago: [The conference] was named 'Evolve' before my time at Cloudera, but now, the word is a perfect fit.
We like to think of the world of data, and by extension AI, in three eras. In the beginning we had the era of control, when everything was in the data center and there were massive platforms like when Cloudera was born [in 2008].
Then the cloud came in, and that made things simple. You could just click a button and lift data onto any hyperscaler. But with that came a lot of optionality. So, even though the infrastructure layer was simpler, the ecosystem became much more complex.
And then the new generation of AI came in four years ago and changed everything. Companies started realizing that they need to use and leverage all their data, not just what they have on the cloud but also what they have on a rack in the office or in their data centers that no one has touched in 15 years. They might have data in Databricks or Snowflake from an acquisition they made. Agents need to be able to access all of that.
What does 'evolve' mean at this moment four years into the AI era?
Gago: Overall, 'evolve' now means that AI is completely transforming the meaning of competitive advantage and internal operations for every human in the world, and companies need to tackle four things: data privacy, control over their systems, the control of their data and having a positive return on investment.
In a competitive landscape, how can Cloudera distinguish itself from competitors to not only serve existing customers but also attract new ones?
Gago: The Cloudera ethos, since the beginning, was based on open source, based on open standards and open tables. The way that translates today is interoperability and optionality. Those are fundamental pieces that separates use from the trend of the last five years that I call 'data hoarding.' Many providers out there have a fantastic product that works really well, but it works well only within their ecosystem -- their specific cloud deployment, their data catalog, their table format. They're very private, closed ecosystems.
In Cloudera, a lot of the components are pliable. We have our own storage service that scales, but we partner with Vast Data, and Dell, and [other vendors] because optionality is an element we believe is fundamental to the era today. We call it the era of convergence. We have our own AI suite … but we just announced a partnership with Mistral. That optionality is a fundamental piece that separates us from many other players.
What more can you say about interoperability?
Gago: The second part is interoperability. We have been investing heavily in Apache Iceberg since its beginning, and that gives us the open table format that most of the world has coalesced on after the open table wars [with Delta Lake] back in the day. We have a few hundred engineers on my engineering team who are contributors to the open-source community. From a catalog perspective, we are investing heavily in Apache Polaris. And, of course, the we components have in our platform are based on many open-source components such as Trino and so on.
[It's important] to give the power back to the user and the owner of the data so they can go wherever they want to go.
If this year there has been greater emphasis on getting value from agents, what do you think will be the big trend in data and AI a year from now?
Gago: I think companies will doble down on efficiency, both operational and financial. We will see a huge evolution of models as well. And agentic workflows will be much deeper in organizations. Operational efficiencies, financial efficiencies and governance of the models are what will be driving the conversation.
Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.
