How agentic RAG supports effective business workflows
AI agents can complete business tasks with little to no human intervention. What if they could access domain-specific knowledge when completing those tasks?
When AI agents have access to multiple, domain-specific data sources and tools, they can make more informed decisions about the next action to take.
To access domain knowledge during workflows, AI agents can use retrieval-augmented generation (RAG). RAG is a search mechanism that retrieves relevant data from a data source and returns it to a large language model (LLM), which then generates a reply to the user. AI agents can apply RAG for task execution and contextual understanding, a capability now often referred to as agentic RAG.
Agentic RAG gives AI agents access to massive, real-time libraries. This enables agents to respond to company-specific prompts or questions, thereby enriching their ability to navigate workflows and achieve goals. In this guide, explore agentic RAG architecture, use cases and a tutorial for setting up agentic RAG with Microsoft Copilot Studio and ServiceNow.
Agentic RAG architecture
Agentic RAG enables an AI agent to use retrieved knowledge as part of its decision-making and action execution. These agents can access multiple data sources and use the knowledge they gain from RAG to help decide which tools to use and what workflows to trigger.
Agentic RAG consists of multiple components. The first component is the agent, which comprises various parts, including tools and data sources. The primary component defined in the agent is the system prompt. The system prompt provides instructions on how the agent should behave. For instance, the agent's instructions could include replying politely or concisely.
An orchestrator manages the agent and ensures that the agent runs and triggers different tasks -- such as tools and knowledge lookup -- depending on the prompt. The orchestrator also integrates with an LLM to process instructions, prompts and provide information to trigger the correct tools or find the right data from knowledge sources.
Agents can have one or multiple sets of data depending on the agent framework in use and the data sources it supports. Data can be anything from a set of single files, Microsoft 365 or other sources.
The description field details what data sources or tools are used for, helping the agent identify which data sources to use for specific prompts and data queries.
For example, consider a user sends a prompt to the agent: "I have issues with wireless connectivity." The prompt goes to the agent, and then, with the applied system prompt, the orchestrator sends this information to the LLM.
Based on the defined instructions and descriptions, the orchestrator triggers a search against knowledge sources to find relevant data. Relevant information found from the RAG source is added to the overall context window, which is a set of data that the LLM has access to.
Agentic RAG examples
One of the most common examples of agentic RAG is a service desk agent configured with multiple data sources to aid troubleshooting tasks. The agent can query data from existing incidents and requests to identify similar issues and create new issues when needed.
RAG can use an internal knowledge base or documentation. For example, ServiceNow could provide knowledge articles for an agent to reference, enabling them to create new incidents and review existing ones.
Other agentic RAG use cases include the following:
- Human resources agent. AI agents access domain data such as company policies, employee handbooks, HR databases and internal communications.
- Financial advisory agent. AI agents access domain data such as financial reports, market data, investment research and regulatory documents.
- Legal research agent. AI agents access domain data such as legal statutes, case law, academic legal articles and court documents.
How to configure agentic RAG
The following example of an IT Service agent with Microsoft Copilot Studio highlights how to configure RAG with agentic AI. While this example focuses on Copilot Studio, many other generative AI agent platforms are available, and the principles and design patterns discussed in this example apply broadly across most platforms.
In this example, a RAG source contained within Microsoft 365 holds information about configuration and troubleshooting related to printer and wireless issues within a company. There is also integration with an IT Service Management (ITSM) tool, which enables users to prompt an agent to create an issue or incident if they are unable to resolve the problem themselves.
To start building the agentic RAG system, first develop instructions that fit the desired business requirements.
This example uses the following instructions:
"Role and Goal: You are a friendly and helpful IT support assistant. Your primary goal is to provide clear, simple and direct answers to user inquiries. You are the first point of contact and should aim to resolve issues efficiently and with a human touch.
Behavioral Guidelines:
- Clarity and Brevity. Your answers must be short, concise and easy to understand. Get straight to the point without unnecessary technical details or jargon.
- Human-like Tone. Communicate in a natural, conversational and friendly manner. Avoid robotic or overly formal language.
- Fact-Based Assistance. Only provide answers you are certain about based on the knowledge you have been given. Do not guess or speculate."
After adding the instructions, create the agent before adding additional components and data sources. Also, ensure that generative orchestration is configured.
Now, add the desired data source, which users can find under Knowledge → Add Knowledge → SharePoint → Browse Items. Select the relevant files.
Because multiple data sources can exist with agentic RAG, it is essential to provide the agent with sufficient context to understand which data source to use when responding to different prompts. Therefore, add a description of the file so the agent knows what kind of information is contained there.
Next, add an integration with an ITSM tool -- this example uses ServiceNow. Provide a set of instructions to give the agent context on when to trigger incidents.
With these configurations set, asking this agent about troubleshooting the wireless network will trigger it to retrieve content from the data source and list it back.
Unable to resolve the incident or need assistance with something else? Ask the agent to create an incident. The agent will automatically use the correct tools to complete the task.
Lastly, monitoring and tracking how agents are used provides insight into how agents make decisions when selecting tools and RAG data sources. Monitoring is often a built-in capability when using cloud-based platforms, but open source frameworks need products like LangSmith or Sentry.io to add the capability.
The future of agentic RAG
Agentic RAG is set to change how we use tech, heading toward a future where AI agents understand what we mean and handle tasks smoothly using everyday language.
The short example highlighted in this article shows how simple agentic RAG can be. I anticipate a growing trend where companies increasingly adopt various AI agents. Some will be user-facing, while others will operate autonomously in the background, addressing issues, managing cases or contributing to business workflows.
While the ecosystem is still young and many of the standards -- including MCP and the A2A Protocol -- have just matured enough to use, many anticipate that agents will eventually replace the need for many SaaS services and provide a more straightforward UI that uses natural language to interact with.
Marius Sandbu is a cloud evangelist for Sopra Steria in Norway who mainly focuses on end-user computing and cloud-native technology.
