What are your recommendations for instituting effective policies and processes for governing sets of big data?
Big data has many definitions, but all of them come down to these main points: It consists of a high volume of material, it comes from many different sources, it comes in a variety of formats, it arrives at high speeds and it requires a combination of analytical or other actions to be performed against it. But at heart, it's still some form of data or content, though slightly different than what has been seen in the past at most organizations. And because it is a form of data or content, business-critical big data needs to be included in data governance processes.
Remember that not all data must be governed. Only data that is of critical importance to an organization's success (involved in decision making, for example) should be governed. For most companies, that translates to about 25% to 30% of all the data that is captured.
What governance best practices apply to big data? The same best practices that apply to standard data governance programs, enlarged to handle the particular aspects of big data:
- Take an enterprise approach to big data governance. All data governance programs should start with a strategic view and be implemented iteratively. Governance of big data is no different.
- Balance the people, processes and technologies involved in big data applications to ensure that they're aligned with the rest of the data governance program. Big data is just another part of enterprise data governance, not a separate program.
- Appoint business data stewards for the areas of your company that are using big data and ensure that they receive the same training as other data stewards do, with special focus on big data deemed necessary due to the technology in use at your organization.
- Include the value of big data governance in the business case for overall data governance.
- Ensure that the metrics that measure the success of your data governance program include those related to big data management capabilities.
- Offer incentives for participating in the data governance program to all parts of the business using big data to encourage full participation from those areas.
- Create data governance policies and standards that include sets of big data and the associated metadata, or that are specific to them, depending on the situation.
Dig Deeper on Big data management
Related Q&A from Anne Marie Smith, Ph.D.
A data lake that isn't well governed may become more of a swamp. Here are key benefits and challenges of data governance in a data lake, plus initial... Continue Reading
A data catalog helps business and analytics users explore data assets, find relevant data and understand what it means. Here are 10 important steps ... Continue Reading
Expert Anne Marie Smith shares five reasons why organizations' analytics programs might fail and how a data management framework and other programs ... Continue Reading