How to keep data silos from damaging your AI projects

What makes AI data silos problematic?

Data silos aren't new in enterprise IT. However, AI has brought new attention to them. The same isolation that stymies data sharing between traditional applications and departments also prevents AI platforms from accessing, training and using siloed data. While silos are a disadvantage to any traditional business infrastructure, they're especially detrimental to AI systems.

The power of AI lies in its ability to establish relationships, find patterns and spot trends, letting it make accurate predictions and sound decisions. This fundamental capability is based on the relationship between different data sets, such as quarterly or annual marketing, sales and manufacturing data. It's the relationship between these varied data sets where nuance and patterns emerge. When an AI system can't access siloed data, it's unable to get a complete picture of the situation and can't deliver accurate analyses and predictions.

Data silos prevent AI from working properly, and the consequences can be significant for the business. They include the following:

• Limited AI accuracy or performance. AI can't make predictions or render decisions at a level the business requires. For example, a sales projection might not be accurate or a medical diagnosis wrong. This quickly translates into poor UX and less user engagement with the AI platform.
• Waste and operational inefficiency. Inaccurate or incomplete AI decisions or recommendations can lead to financial and material expenditures that don't translate into revenue or enhance business efficiency.
• Stagnant innovation. Because data silos prevent a full picture of a situation, missing data can cause an AI system to miss relationships or opportunities that might otherwise be found, creating a serious roadblock to business innovation.
• AI platform failure or abandonment. In extreme cases, consistently poor UX might adversely affect the organization's reputation. This could lead a business to stop using or putting resources into maintaining an AI platform.

Causes of AI data silos

Why do data silos still exist, given the sophistication and importance of AI for modern businesses? The reasons are varied, but they follow many traditional causes that have gone unresolved due to rapidly changing business and technology environments. The most important causes of AI data silos include the following:

• Unclear business leadership. Technological mastery is vital for modern business, but it's hardly quick or easy. The road to technical success starts at the top with clear leadership and direction. Data silos thrive where AI vision is lacking and when leadership doesn't properly address them before starting an AI project.
• Technical issues. Businesses often yield to the path of least resistance to quickly implement tools, platforms and systems, leading to interoperability and integration problems. When systems don't communicate well, data silos form, trapping data that AI projects can't readily access.
• Diverging organizational goals. Managers often make technical decisions that are well-suited to their department's needs but don't prioritize broader interoperability with other business units' data. A manufacturing department might collect and retain data in radically different detail and formats than sales and marketing departments without regard for each other's data practices. This quickly leads to technical issues that perpetuate data silos detrimental to AI initiatives.
• Regulatory demands. Differing approaches to data governance, data quality, data retention and regulatory compliance can lead to divergent data management practices across business units. For example, data stores containing highly sensitive personally identifiable information (PII) might be isolated from other systems by design or intent -- the need for security often takes precedence over data interoperability.
• Business growth. Data silos can arise as the business grows and changes, and new technologies are brought in. For example, a merger might force the acquired business unit to adopt its parent company's tools and data standards, leaving older data siloed and less accessible.

How to find AI data silos

Finding AI data silos is a challenge. Each siloed instance might work perfectly for its associated workflow, but difficulties arise when an AI system tries to use multiple data sources together. These best practices can help AI teams identify data silos that might be affecting AI systems:

• Inventory AI data sources. This typically involves a detailed audit of all applications and data sources in use, correlating which departments or business units use specific applications and corresponding data sources. Knowing what's there is key to determining what the AI system can use and how readily it can be accessed. This often requires IT support. In many cases, AI teams will discover overlooked enterprise applications and data stores.
• Find isolated applications. Modern businesses use centralized data systems that are far more versatile and interoperable than local workspace data, such as an Excel spreadsheet or SQL database. When local applications are used within a business workflow, a data silo is almost guaranteed, and an AI system is unlikely to be able to access that data.
• Look for duplicated data. Isolated applications might use varied copies of data that can exist in different formats in different business units. These duplicate data stores can quickly fall out of sync, resulting in different content for the same entries or references. Using duplicate data can confuse an AI system, so it's important to reconcile duplicate data before AI uses it.
• Check data metrics. It's common for different applications and unique data stores to generate similar business information. For example, sales and finance teams might use their own unique data stores and applications to calculate common metrics, such as quarterly marketing budgets. Reconcile these data silos to avoid applications reporting different results for the same query.
• Identify data access delays. Multiple applications can access centralized data sources throughout the workflow with little, if any, delay. If data access is delayed because the data must first be requested, reformatted or transferred between stores -- especially if this manipulation is a manual effort -- a data silo is present. AI can't operate effectively in the face of these sorts of delays, so they should be remedied before AI accesses the data.
• Consider system integration and compatibility. Data silos can arise from poor system integration and interoperability, such as vital legacy systems operating alongside state-of-the-art enterprise platforms. These disconnects often create data silos that AI systems can't effectively use. Business leaders might need to invest in new technologies, such as updating legacy systems to use a common data source, before AI teams can fully use the data.
• Recognize data access restrictions. Some data stores might be deliberately inaccessible because of security measures that restrict access to sensitive data or PII. These data stores might not be made available to AI projects without safeguards, such as data anonymization or synthetic data generation. Proper access to restricted data requires careful collaboration between IT, business, AI and governance teams.

Overcoming AI data silos

Finding AI data silos is one challenge. Fixing those data silos for adequate AI access is another problem entirely. Once AI data silos are identified, there are several strategies for correcting them:

• Modernize data storage technologies. Data lakes and data warehouses can support centralized data storage for both structured and unstructured data. This offers a single repository that AI platforms can access and process. Other technological choices can include a data fabric environment that uses a virtual data management layer to create a unified data view while enabling data to remain in its original storage locations. Similarly, APIs can enable disparate systems to communicate and exchange data without modifying or migrating the information.
• Perform regular data storage inventories. Data inventories aren't a one-time tactic; they should be a regular practice that can prevent new data silos from emerging over time. Regular inventories make it easier to find, clean and delete duplicate or outdated data. This helps maintain high levels of data quality for AI training and optimization.
• Use data management tools. The use of data management tools ensures data quality and keeps data organized for AI training. For example, extract, transform and load (ETL) tools can be used to automatically handle data extraction, transformation and loading to keep AI data prepared and organized. As another example, data management tools can be used to establish and maintain shared data structures. This helps ensure consistent, properly structured data for AI training.
• Implement comprehensive data governance. Create, deploy and regularly update data governance policies that apply across the business. This defines data ownership and lines of data responsibility. It also sets guidelines that define data access, change and sharing among users and business units. It requires careful leadership and regulatory clarity to balance compliance with data interoperability for AI projects.

Stephen J. Bigelow, senior technology editor at TechTarget, has more than 30 years of technical writing experience in the PC and technology industry.