Evaluating agentic AI for demand forecasting

What agentic AI means in a forecasting context

The notion of agentic AI is somewhat subject to interpretation across different vendor descriptions. For our purposes, agentic AI pertaining to demand forecasting has three basic properties:

• Autonomy. The agent must make choices and perform tasks without any user input at each step. This might include obtaining external signals, changing system parameters and prompting the system to start reordering.
• Goal-driven reasoning. Actions taken by the AI system depend on predetermined goals, such as lowering forecast errors or improving inventory turnover, while the specific actions depend on the situation.
• Awareness of surroundings. The system must be able to understand and react to various environmental signals, such as sales results, forecasts, social media trends and global economic developments.

4 evaluation dimensions CIOs must prioritize

In its 2022 research on the use of AI in company supply chains, McKinsey & Company outlined the potential of AI demand sensing to reduce forecast error relative to traditional forecasting by 20% to 50%. However, realizing this potential is not just about CIOs choosing a suitable platform to run AI demand sensing on; CIOs need to make a number of subtler decisions.

1. Data readiness and signal architecture

The effectiveness of agentic AI depends on the environment's signal quality. To evaluate any platform, perform an honest audit that covers the following data layers to determine where you might need more data integration work:

• Internal structured data. This includes ERP transactional histories, web map service feeds, promotional calendars and pricing data. Most companies already have some internal structured data sources, though they're often fragmented across silos that are difficult to govern.
• External structured data. This refers to syndicated market data (e.g., competitor price lists), weather APIs, economic indices and supplier lead-time feeds. Complexity in integrating with external sources is usually the largest barrier for most companies.
• Unstructured data. News feeds, social media trends, earnings call transcripts and port congestion reports are unstructured signals that need natural language processing pipelines prior to being processed by a forecasting engine.

If an organization hasn't yet implemented comprehensive data integration strategies -- and that's most businesses right now -- its journey to evaluate and implement agentic AI systems begins not with model selection but with data infrastructure investments.

Thus, CIOs must develop a signal architecture roadmap before issuing a request for proposal to potential AI vendors, which often overpromise their ability to work with existing data infrastructures.

2. Human-in-the-loop design and explainability

As agentic systems become more autonomous, governance can become a challenge. CIOs are tasked with ensuring adequate human governance over systems where forecast errors are possible. This can be significantly more difficult with decreased human input.

For example, the EU AI Act ranks AI applications in supply chains by risk level. High-risk systems must meet requirements for explainability and human oversight. Even when these requirements aren't mandated for a particular application, building in checkpoints for human review is essential to catch and audit erroneous forecasts before they affect operations.

To do this, CIOs should design three levels of governance.

• Low-risk decisions. The model is designed to make the most common, lowest-risk decisions on its own. The agent can adjust safety stock levels for fast-moving SKUs daily, without any human input.
• Medium-risk decisions. The agent makes the changes but alerts humans for review. For example, if demand forecast adjustments exceed predefined thresholds, the system implements the changes while notifying planners to verify the modifications.
• High-risk decisions. Humans must approve forecasts or recommendations before implementation. The agent can recommend actions for high-value procurements or new product launches, but humans must approve those recommendations.

Additionally, CIOs should make model explainability a non-negotiable requirement. When evaluating vendors, ask them to demonstrate the system's ability to explain a forecast variance using real-world examples, not simply using a prepared demonstration.

3. Integration architecture and legacy system compatibility

Most enterprise demand forecasting processes are characterized by a fractured set of ERP systems and planning tools acquired in numerous M&A deals. Agentic AI systems aren't an exception in this regard, offering little in the way of simplification to the existing environment of planning algorithms, collaborative planning communities and planning analytics.

Ask the following questions when shortlisting agentic AI vendors:

• Does the agentic system rely on out-of-the-box ERP connectors or use proprietary APIs to ingest the data?
• How does the agentic system account for data latency -- i.e., the delay between transactional data and its appearance in analytic data warehouses accessible to the forecasting algorithms?
• How does the agentic system reverse commissioning errors, which lead to faulty forecasts already deployed in the operational systems?

4. Total cost of ownership and value measurement

In practice, integration costs often exceed the costs of the agentic AI software itself.

Agentic AI platforms have a different cost framework than traditional forecasting tools. In addition to the costs of license agreements, CIOs need to consider the following:

• Compute costs. For example, agentic systems that process large amounts of data as it's generated by external sources can be expensive in terms of computing resources. This is especially true if they use cloud-based services that charge based on usage.
• Model maintenance. An agentic system doesn't rely on a static statistical model that requires periodic updates. Instead, it evolves over time through changes in its prompts and through continued prompt tuning to improve performance. These activities are typically resource-intensive, requiring an individual with expertise in both machine learning operations and enterprise architecture.
• Change management. From an operational standpoint, the transition from planners who drive forecasting processes to the use of AI as a tool to support forecasting is significant. Implementation plans often underestimate the resistance to this change from members of the demand planning organization.

Traditional forecasting metrics focus on forecast accuracy alone. Instead, businesses should measure the quality of AI-driven decisions and their effect on inventory outcomes. The real value lies in using forecasts to optimize inventory positioning, reduce stockouts and minimize excess and obsolescence costs.

A phased implementation approach

These scenarios represent gradual agentic AI adoption and transformation from manual processes to automation. The various aspects of AI should be rolled out in different phases to enable the growth of experience, measurement, performance validation and appropriate governance.

• Phase 1: Contained pilot (months one to six). The contained pilot typically starts with a single product category or business unit that has good data and stable demand patterns. The organization can then develop key metrics for forecasting within that business and put the agentic AI in shadow mode or run it side by side with the current process. After several months, the organization can compare results to the current process and see if they are significantly better.
• Phase 2: Supervised deployment (months seven to 12). In this phase, the AI operates autonomously but with human oversight. As you integrate more external data into the model, humans evaluate how the AI supports business decisions and measure actual business outcomes -- not just forecast accuracy. In this phase, document the AI's functionality and establish the appropriate governance.
• Phase 3: Scaled rollout across organization (month 13+). With enough experience from the pilot, the AI agents can be scaled to other product categories and across the organization. Formalize lessons learned into a human-in-the-loop governance framework with clearly defined roles and responsibilities. This framework should integrate into the broader enterprise AI governance strategy developed throughout earlier phases. The organization can now accelerate agentic AI adoption while managing operational risks.

Attempting to bypass a formal pilot and proceed directly to governance and integration at scale quickly becomes an expensive problem.

The CIO's bottom line

Agentic AI for demand forecasting is a fundamental change in functionality, not just a new software tool. Organizations seeking to realize its full potential must transform their current processes -- this isn't simply buying a new technology.

CIOs can use this framework to evaluate both the technology requirements and organizational changes needed before committing resources. Agentic AI is ready for prime time; it is the data infrastructure, the governance and the organizational culture that need to be brought along.

Kishan Kumar is a buyer at Pearse Bertram+ based in Bloomfield, CT. In his nine-year career, he has worked on strategic sourcing, procurement and supply chain management within the manufacturing, industrial automation and FMCG industries. Kishan earned his MBA from Southern Connecticut State University and his Supply Chain Strategy & Management certification from the Indian Institute of Management. Kishan is a certified Six Sigma Green Belt, SAP Certified Implementation Consultant, Agile Leader Certified professional and holds a Generative AI for Project Managers certification from PMI. Kishan is currently an active professional member of the Association for Supply Chain Management (ASCM) and has written articles that were published on SCMR.com