New Databricks tool aims to up agentic AI response accuracy
With enterprises struggling to successfully develop agents using traditional RAG pipelines, the vendor's new Instructed Retriever offers an alternative.
With the launch of Instructed Retriever, Databricks is attempting to help customers more successfully develop agents and other AI applications.
Sparked by OpenAI's November 2022 launch of ChatGPT, many enterprises have significantly increased their investments in developing AI tools such as chatbots and agents that can make their workers better informed and their businesses more efficient.
In response, given that data is what gives AI applications knowledge, data management and analytics vendors created environments within their platforms designed to simplify the complex process of building secure pipelines that continually feed data to applications to keep them accurate and relevant.
San Francisco-based Databricks, one of the pioneers of the data lakehouse format for storing data, has been one of the more aggressive builders of a development environment for generative AI (GenAI) and agentic AI tools. Databricks rival Snowflake has similarly made AI development a focus, as have hyperscale cloud vendors such as AWS, Google Cloud and Microsoft along with scores of smaller vendors specializing in data and analytics.
Struggling to succeed
Despite heightened interest in AI over the past three-plus years and attempts to simplify building AI tools, the overwhelming majority of AI projects never make it past the pilot stage and into production.
Traditional RAG treats retrieval as a stateless keyword-matching step and can lose the user's intent and constraints before reaching the generation phase. Instructed Retriever seems to be a practical solution to solve the problem [by taking] RAG beyond simple prompt engineering.
Sanjeev Mohan Founder and principal, SanjMo
One of the primary problems preventing more successful AI development is that it's difficult to train agents and other AI tools to retrieve the right data to respond to a user's query. Applications that can't deliver an acceptable rate of relevant, accurate responses are useless to a business.
Perhaps the most common means of connecting AI applications with source data is retrieval-augmented generation (RAG). But as evidenced by the high failure rate of AI initiatives, it hasn't been good enough.
Instructed Retriever is a new alternative that adds more context to data retrieval than traditional RAG systems and outperformed RAG in benchmark tests conducted by Databricks. As a result, it is a significant addition to the vendor's Agent Bricks suite for agentic AI development, according to Sanjeev Mohan, founder and principal of analyst firm SanjMo.
"Traditional RAG treats retrieval as a stateless keyword-matching step and can lose the user's intent and constraints before reaching the generation phase," he said. "Instructed Retriever seems to be a practical solution to solve the problem [by taking] RAG beyond simple prompt engineering."
If Instructed Retriever improves upon traditional RAG as intended, it will force competing vendors such as Snowflake and hyperscale cloud providers AWS, Google Cloud and Microsoft to respond, Mohan continued.
"If the performance numbers are indeed substantial, expect other vendors to quickly follow Databricks' lead," he said.
Instructed Retriever differs from RAG in the way it translates user intent into precise search queries and delivers responses, according to a Databricks blog posted on Jan. 6 that introduced the new AI development tool.
RAG systems often use only a user's query to generate a response, not even adhering to specific instructions such as, "Find documents only from last year." Instructed Retriever, conversely, augments a query with additional specifications such as user instructions, previous examples and indexed knowledge source schemas.
The result, according to Databricks, is substantially improved performance over RAG, which makes Instructed Retriever a valuable new feature in the vendor's agentic AI development suite, according to Kevin Petrie, an analyst at BARC U.S.
"Instructed Retriever is a solid step forward for Databricks," he said. "AI adopters have embraced RAG as the ideal method of applying powerful GenAI models to their proprietary data, but they still struggle to ensure that GenAI agents have the right data and contextual metadata on hand to answer a given prompt. This new offering helps close that gap with schema-aware instructions and iterative searches."
However, the cost of cloud computing is a concern for many enterprises, so Databricks customers should be wary of any expenses using Instructed Retriever might add to their AI development efforts, Petrie continued.
"Databricks users should keep an eye on the implications for compute costs because these new capabilities add processing overhead," he said. "AI adopters are increasingly concerned about the cost of their initiatives, in particular for software, compute and token consumption."
Customer feedback provided Databricks with part of the impetus for developing Instructed Retriever, according to Michael Bendersky, the vendor's director of research.
He noted that most AI vendors provide some form of retrieval to give agents enough context to respond to user queries and properly perform their tasks. However, many RAG systems focus only on what to retrieve rather than how to retrieve rather than how to do so. Databricks' new tool is designed to augment what data to retrieve for AI agents with how to retrieve it.
"With Agent Bricks, we've heard a lot of feedback from customers that ask questions that require complex filtering or result boosting, which motivated us to create a smarter retriever," Bendersky said.
In addition to user instructions, previous examples and indexed schemas, Databricks included the following capabilities to Instructed Retriever to improve the retrieval process:
Query decomposition to enable agents to break down complex, multi-part requests into a full search plan.
Contextual relevance including a ranking feature to best match a user's intent.
Metadata reasoning to translate natural language queries into precise search filters.
Mohan noted that query decomposition, context-based ranking and metadata filtering are commonly used in search systems. They are not, however, common in connecting agents with models and other data sources.
"It is impressive to see them being used for LLM-based retrieval and making [retrieval systems] instruction-aware," Mohan said.
Next steps
Instructed Retrieval is just one tool in Agent Bricks aimed at helping Databricks customers more successfully build AI agents. In November, the vendor added agent quality and agent observability capabilities to Agent Bricks.
One future focus will be better enabling users to combine structured and unstructured data to inform agents, according to Bendersky.
"We're continuing to focus on helping our customers in building agents that provide grounded, high-quality answers based on all of their data," he said. "Sometimes the answer requires combining many types of data -- structured and unstructured -- which remains a complex challenge today.
One way that Databricks might help customers' agentic AI development effort is by adding capabilities -- perhaps through acquisition -- that better enable developers to proactively address potential problems with data to ensure accurate data inputs, according to Petrie.
"While Databricks offers some capabilities in this area, such as anomaly detection and data profiling, they might want to take a bigger step by acquiring an expert vendor in this space," he said, noting that BARC research shows that data quality is a top concern during AI development. "Their partner Anomalo might be a good acquisition to consider as they address both structured and now unstructured data."
Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than 25 years of experience. He covers analytics and data management.
