Data quality gaps undermine the promise of agentic NetOps
EMA's NetOps survey finds only 44% trust their network data for AI. Packet and config data create major risk, so teams must standardize, invest and audit.
Whether people believe in AI or not, nearly everyone agrees that AI requires high-quality data.
AI models must have access to high-quality data for training and inference. As IT organizations apply AI to infrastructure and operations, they find that bad data can shatter their AI hopes and dreams.
Enterprises Management Associates recently found hard evidence that supports this truism. We surveyed 458 IT professionals about their efforts to apply AI to network operations. Only 44% were confident that the quality of their network data would support these efforts.
This confidence gap should not be ignored. EMA research found that low confidence in network data correlated with lower rates of the following criteria:
User adoption of AI tools.
User trust in AI tools.
ROI expectations for AI tools.
Rates of overall success with AI-driven NetOps.
How packets and config data limit AI
IT organizations generally maintain a wide variety of data about their networks. In EMA's experience, these data stores are generally disorganized. They are siloed, often proprietary and sometimes of dubious quality. However, certain classes of data are particularly challenging to AI-driven NetOps ambitions -- packets and config data.
A deep analysis of EMA survey data revealed that data confidence is particularly low in enterprises that consider configuration data and packets as essential to their AI-driven NetOps projects. In other words, a strategic focus on config and packet data leads to failure.
EMA suspects that this is a data quality issue. IT organizations must improve their approach to acquiring, managing and using this data.
Do better than ad hoc packet capture
Many organizations take an ad hoc approach to packet capture. They collect packets only in reaction to events that require deep analysis. This data capture might be triggered by an event or engineers might manually capture the data. In either case, it leads to an incomplete record of traffic.
Continuous packet capture is a better approach. It should also be done systematically, where packets are gathered from strategic points on the networks. Ad hoc capture typically occurs in one location relevant to a specific investigation.
Furthermore, packets are expensive to capture and store. Network teams should consider using metadata derived from these packets. AI tools need access to raw packets for forensic analysis, but packet metadata is valuable across a variety of AI use cases. Network teams should use packet visibility tools that can generate rich metadata.
Adopt a source of truth for configs
As an analyst who focuses on network infrastructure and operations, I have heard many horror stories about config data chaos.
If you have a multi-vendor network and use multiple network tools to manage it, I guarantee you have a data problem.
A network architect at a global pharmaceutical company once told me that none of his 1,000 routers were compliant with his company's config standards. Each router had unique deviations from the golden config. He had a network automation tool that could fix the issue, but he wasn't allowed to push a global change, so the routers remained 1,000 unique snowflakes.
Another network architect told me his company didn't have any configuration standards on its network. When it was time for him to establish a source of truth for his network configuration, he ran a discovery tool that pulled config files from every device. His assumption was that the network's state was the intended standard. However, the next step of his process was to validate the config reality against business application priorities, security policies and compliance requirements. He quickly discovered that the product network was not aligned with any of these requirements.
One theme across the two examples above is complexity. Establishing a good data store of config information is a long and painful project, but it must be done if config data is strategic to your goals for agentic operations.
Next steps for agentic AI data management
Agentic NetOps has a data problem. If you have a multi-vendor network and use multiple network tools to manage it, I guarantee you have a data problem. It might not be purely about packets or config data -- the landmines could be anywhere, and you're probably already aware of many of them.
My analysis of EMA's survey data revealed five steps network professionals can take to correct this course.
Know why you're adopting AI. EMA data shows a clear correlation between understanding the business value of AI-driven NetOps and confidence in network data. Know why you're doing this. Get buy-in from the data owners and ensure the data is correct.
Spend. Budget limitations correlate strongly with data uncertainty. Your organization will need to invest in tools and expertise to discover, audit, clean and organize this data.
Establish or work with an AI center of excellence. Knowing how to evaluate AI tools is essential to success, and confidence in evaluation capabilities correlates strongly with data confidence. As you evaluate tools, you'll see how they work with network data.
Demand openness, support of industry standards and deep integration from vendors. Data quality problems hide in the dark. Integrations and open standards light up that darkness.
Request AI vendors offer tools to continuously audit and verify AI outcomes. This builds trust in AI tools, but it should also help you identify any data quality drift.
Shamus McGillicuddy is vice president of research for the network management practice at Enterprise Management Associates (EMA). He has more than 20 years of experience in the IT industry and has written extensively about the network infrastructure market. Prior to joining EMA, McGillicuddy was the news director for TechTarget's networking site.
