Data quality techniques and best practices
Top Stories
-
Tip
28 Nov 2022
5 steps to an improved data quality assurance plan
Follow these steps to develop a data quality assurance plan and management strategy that can help identify data errors before they cause big business problems. Continue Reading
-
Tip
17 Oct 2022
Make data usability a priority on data quality for big data
To help make big data analytics applications more effective, IT teams must augment conventional data quality processes with measures aimed at improving data usability for analysts. Continue Reading
-
Tip
28 Jul 2022
What is data lineage? Techniques, best practices and tools
Organizations can bolster data governance efforts by tracking the lineage of data in their systems. Get advice on how to do so and how data lineage tools can help. Continue Reading
-
Answer
07 Jun 2022
Data lake governance: Benefits, challenges and getting started
A data lake that isn't well governed may become more of a swamp. Here are key benefits and challenges of data governance in a data lake, plus initial steps to take. Continue Reading
-
Answer
01 Jun 2022
How to build a data catalog: 10 key steps
A data catalog helps business and analytics users explore data assets, find relevant data and understand what it means. Here are 10 important steps for building one. Continue Reading
-
Tip
26 May 2022
Top benefits of data governance for businesses
Effective data governance provides a variety of benefits to organizations, including improvements in operational efficiency, data quality and business decision-making. Continue Reading
-
Feature
06 May 2022
8 proactive steps to improve data quality
Here are eight steps to take to improve your organization's data quality in a proactive way, before data errors and other issues cause business problems. Continue Reading
-
News
10 Feb 2022
Data quality startup Superconductive raises $40M
The vendor behind the open source Great Expectations data quality technology is looking to build a commercial cloud service to help data professionals accurately profile data. Continue Reading
-
Feature
02 Feb 2022
Self-service data preparation: What it is and how it helps users
Using self-service tools to properly prepare data simplifies analytics and visualization tasks for business users and speeds complex modeling processes for data scientists. Continue Reading
-
Feature
31 Jan 2022
6 data preparation best practices for analytics applications
Amid issues of data sources, data silos and data quality, the process of collecting and prepping data for analytics applications requires a practical and effective approach. Continue Reading
-
Feature
27 Jan 2022
Data preparation in machine learning: 6 key steps
Trustworthy analytics outcomes depend on the right data, requiring data scientists to focus on these steps when they prepare data for use in machine learning applications. Continue Reading
-
News
10 Jan 2022
Anomalo brings data quality platform to Snowflake
Elliot Shmukler, co-founder and CEO of Anomalo, details how he was inspired to build his data startup after a critical data failure while he was working at Instacart. Continue Reading
-
Feature
05 Jan 2022
Western Union looks to improve data quality with Talend
Thomas Mazzaferro, chief data officer at Western Union, outlines the financial services company's approach to cloud data migration and how Talend fits into the data architecture. Continue Reading
-
Feature
17 Dec 2021
Data forecast for 2022: Data quality and cloud convergence
Expect data quality to be a top area of investment and activity next year as the need to trust data for operations, insight and machine learning will only continue to grow. Continue Reading
-
News
16 Dec 2021
Bigeye brings new visibility to data observability
The startup continues to build out its cloud platform that enables data engineers to identify and remediate data pipeline quality and availability problems. Continue Reading
-
News
19 Nov 2021
LogicMonitor introduces centralized log management platform
The data observability platform vendor's new platform enables enterprises to use AIOps and automation to find anomalies with their IT system, report them and head off widespread problems. Continue Reading
-
Tip
11 Nov 2021
Business shift to a data monetization strategy elevates CDOs
As their focus dramatically swings from compliance issues to data monetization, chief data officers are on track to take their rightful place among C-level executives, but slowly. Continue Reading
-
News
09 Nov 2021
Datafold raises $20M for data reliability engineering
Datafold's founder and CEO details the data observability challenges the startup is looking to address with its suite of data tools that provide visibility into data pipelines. Continue Reading
-
Feature
08 Nov 2021
Knowing the right data enrichment techniques is crucial
In 'The Enrichment Game,' the latest book by 'Data Guy' Doug Needham, you'll learn why and how to cull data to gain valuable insights and a business advantage. Continue Reading
-
Feature
02 Sep 2021
How to choose exactly the right data story for your audience
A data practitioner has two jobs: tell the right data story and in the right way to win over project stakeholders, data expert Larry Burns says in his latest book. Continue Reading
-
Feature
01 Jul 2021
EY CTO outlines data governance challenges
Multinational professional services firm EY has taken a strategic view of how to manage and use data in a federated approach powered by a trusted data fabric. Continue Reading
-
Feature
18 Jun 2021
How data governance and data quality work together
High-quality, reliable data is essential to the data governance process. Here are strategies to ensure data quality standards are ingrained in governance processes. Continue Reading
-
News
10 Jun 2021
Quest updates Erwin Data Modeling and Data Intelligence
Quest is looking to build out what it calls a data empowerment platform with the help of data modeling and other technology from its recently acquired Erwin business unit. Continue Reading
-
Feature
10 Jun 2021
The enterprise advantages of automated data collection
Many organizations still rely on manual data entry that wastes time and results in low-quality data. Here are the latest automated data collection techniques and their benefits. Continue Reading
-
News
10 Jun 2021
Prefect raises $32M for dataflow automation technology
Data startup Prefect is looking to grow its open source project and commercial business to help organizations manage automated data pipeline requirements. Continue Reading
-
News
03 Jun 2021
Cribl aims to ease data observability with LogStream update
LogStream 3.0 brings new configuration capabilities to Cribl's pipeline technology that can help organizations optimize log and metrics data. Continue Reading
-
Feature
01 Jun 2021
How automated metadata management improves business insights
Automating metadata management can cut down time spent on tasks such as data tagging and cataloging. Explore how automated metadata management is improving data quality. Continue Reading
-
Feature
29 Apr 2021
Building a big data architecture: Core components, best practices
To process the infinite volume and variety of data collected from multiple sources, most enterprises need to get with the program and build a multilayered big data architecture. Continue Reading
-
Feature
27 Apr 2021
Data quality for big data: Why it's a must and how to improve it
As data volumes increase exponentially, methods to improve and ensure big data quality are critical in making accurate, effective and trusted business decisions. Continue Reading
-
Feature
15 Apr 2021
Who belongs on a high-performance data governance team?
Putting together a high-quality data governance team can be a challenge. Explore the necessary team members and best practices for a high-performing team. Continue Reading
-
Tip
08 Apr 2021
IoT data quality determines project profitability
The insights drawn from IoT data analysis can only improve with better IoT data quality, which data scientists measure with metrics such as accuracy, timeliness and completeness. Continue Reading
-
Feature
18 Feb 2021
Why your data story matters and how to tell it
Data storytelling isn't just for business analysts. Find out how to build a data management story and why you need to have one in the first place. Continue Reading
-
News
03 Feb 2021
Collibra acquires predictive data quality vendor OwlDQ
With OwlDQ, Collibra is bringing data quality capabilities to its data intelligence platform to help users better understand and evaluate the impact of different sources of data. Continue Reading
-
Feature
28 Jan 2021
Pandemic exposes difficulty of data management in education
Limited resources and a shift to remote learning have shown the inequalities across school districts when it comes to data management and the negative impact this can have. Continue Reading
-
Tip
29 Dec 2020
Data lineage documentation imperative to data quality
Understanding the detailed journey of data elements throughout the data pipeline can help an enterprise maintain data quality and improve trustworthiness. Continue Reading
-
Feature
10 Dec 2020
The top 6 use cases for a data fabric architecture
Enterprise data fabric adoption has been on the rise as a way to ensure access and data sharing in a distributed environment. Here are the top use cases for data fabrics. Continue Reading
-
Feature
13 Nov 2020
Data protection officer responsibilities and role importance
The data protection officer title has been growing over the last few years, and organizations are still working to grasp the full responsibilities of the role. Continue Reading
-
Feature
16 Oct 2020
How AI data privacy can help your enterprise
Enterprises benefit in many ways from AI data privacy tools that reduce the need for manual efforts from data professionals. Read on for top use cases for the growing technology. Continue Reading
-
News
16 Oct 2020
The new normal for enterprise data governance
Mastercard data exec highlights the foundational role of enterprise data governance during the pandemic era with more people working from home and new demands on businesses. Continue Reading
-
Feature
25 Sep 2020
Key steps in the feature engineering process
Feature engineering is key to machine learning algorithms. Read on to learn how those features are created and chosen to increase the accuracy of those models. Continue Reading
-
Feature
24 Sep 2020
Talend evolves enterprise data integration efforts
The CTO of Talend outlines the path the data integration vendor has taken in recent years as the market for big data has evolved along with the need to treat data as an asset. Continue Reading
-
Feature
15 Sep 2020
The evolving role of the chief data officer
The job of the chief data officer is expanding to be more strategic as the need for organizations to connect and make sense of vast sums of data continues to grow. Continue Reading
-
Feature
24 Aug 2020
Why you should consider a machine learning data catalog
A machine learning data catalog can benefit an enterprise in a variety of ways, from increasing access to necessary data to keeping your data sources up to date. Continue Reading
-
Feature
14 Aug 2020
How to streamline your data cleansing process
Data cleansing is an important part of maintaining data quality, and the process is easier if you keep ahead of it by upholding governance and quality standards. Continue Reading
-
Feature
07 Aug 2020
Top 7 data catalog use cases for enterprises
From data lake modernization to data democratization, there are many benefits to a data catalog. Experts talk about the top ways data catalog adoption can benefit enterprises. Continue Reading
-
Feature
17 Jul 2020
Top 5 feature engineering tips for better models
From understanding a model's expected goal to factoring in subject matter expertise, experts talk about the best ways to improve your feature engineering. Continue Reading
-
Feature
30 Jun 2020
Surprising insight on the data governance process
Many enterprise data governance processes have been around for a while. Read on for ways a modern approach can be beneficial, and preview a new book that can update your strategy. Continue Reading
-
Feature
29 May 2020
GDPR, CCPA, cloud drive security management tool makeovers
As data protection and privacy laws like GDPR and CCPA take hold, data managers refine governance practices, while vendors enhance traditional big data security tools. Continue Reading
-
Feature
29 May 2020
Improving enterprise data governance with data trust
Talend users from Covanta and AutoZone provide insight into data management challenges amid growing data complexity at the Talend Connect virtual conference. Continue Reading
-
Feature
18 May 2020
Alternative data in finance holds potentials and risks
Alternative data sources, when verified as reliable and trustworthy, can be an essential tool for financial institutions when it comes to gathering information. Continue Reading
-
Feature
27 Apr 2020
Data governance and COVID-19 data security challenges
Maintaining data governance and data security best practices is essential now more than ever. But the increase in working from home can put a strain on those practices. Continue Reading
-
Feature
17 Apr 2020
Common data lake challenges and how to overcome them
Managing the data contained in your enterprise data lake presents many challenges. From the amount of data to data inconsistencies, here are some solutions to common issues. Continue Reading
-
Feature
24 Mar 2020
How data governance and data management work together
Data governance is an important part of data management. These recommendations will help build a strong complementary relationship between the two strategies. Continue Reading
-
Feature
16 Mar 2020
How to build an effective streaming data architecture
Data architecture can be tricky when it comes to real-time analytics. Clear objectives and scalability are important factors when determining the streaming data architecture you need. Continue Reading
-
Feature
12 Feb 2020
Data governance metrics: Data quality, data literacy and more
Statistics on data quality are the most common metrics used to track data governance programs, but other types of data-related measurements also can be applied. Continue Reading
-
Tip
10 Feb 2020
6 best practices on data governance for big data environments
Efforts to govern big data must corral a mix of structured and unstructured data. That's a challenge for most organizations. These six action items will help. Continue Reading
-
Feature
06 Feb 2020
Breaking down data silos with strong data governance
Not having a single source of truth can be a huge issue for data professionals. But strong data governance policies can prevent data silos and lead to better-quality data. Continue Reading
-
News
23 Jan 2020
Hitachi Vantara acquires data catalog vendor Waterline Data
With the acquisition of Waterline Data, Hitachi Vantara is bringing new data catalog capabilities that will expand the Lumada Data Services and DataOps portfolio. Continue Reading
-
Answer
21 Jan 2020
What are the main features of data catalog software?
Data catalogs are curated data portals for self-service analytics users. Learn about the capabilities of data catalog tools, vendors that sell them and how catalogs work. Continue Reading
-
Feature
08 Jan 2020
Evaluate your data governance and privacy best practices
GDPR and CCPA have put data governance in the news. With this increase in data legislation, now is a great time to review your company's data governance policies. Continue Reading
-
Feature
26 Dec 2019
How to improve data governance for self-service analytics
Citizen data scientists and self-service analytics are on the rise as the data scientist shortage continues. Here are some data management best practices to support them. Continue Reading
-
Feature
17 Dec 2019
Data catalog management for analytics fraught with unique demands
Data catalogs for analytics applications demand detailed assessment of user needs, cross-functional teams, ready access, continuous improvement and a self-sustaining system. Continue Reading
-
Feature
10 Dec 2019
How to streamline feature engineering for machine learning
Structured data is necessary in machine learning, but sifting through data is time consuming. Streamlining the feature engineering process can help data scientists be more productive. Continue Reading
-
Feature
14 Nov 2019
How AI could save the enterprise data catalog
Enterprise data catalogs record all the databases and files in a corporation, but sometimes it can become out of date. Advances in AI are helping minimize the problem. Continue Reading
-
Feature
10 Oct 2019
DOD CDO shares 7 data management best practices
The DOD's first chief data officer, Michael Conlin, shares tips on how organizations can modernize their data management practices with the objective of matching private enterprise. Continue Reading
-
Feature
04 Oct 2019
Align business and IT drivers through data quality best practices
Information management consultants talk about the importance of aligning business and IT drivers for a holistic architectural approach to ensuring strong data quality. Continue Reading
-
Feature
30 Jul 2019
Information Builders exec talks data management ethics
In this Q&A, James Cotton, director of the Data Management Centre of Excellence at Information Builders, offers advice on how organizations can carry out ethical data management. Continue Reading
-
Tip
29 Jul 2019
How to navigate today's business analytics governance challenges
Don't let a traditional analytics mindset lure you into complacency when it comes to advanced analytics governance. Here are the biggest governance roadblocks and how to avoid them. Continue Reading
-
Feature
24 Jul 2019
Key components of an effective data virtualization architecture
Experts break down which elements -- both technical and nontechnical -- are most crucial to successfully deploying and managing a data virtualization architecture. Continue Reading
-
Tip
28 May 2019
The evolution of the data preparation process and market
Organizations have long struggled with inconsistent data and other issues. Expert Andy Hayler explores how that has led to the rise of the data preparation tools market. Continue Reading
-
Feature
14 May 2019
Advice on enterprise data cleansing from an SAP VP
SAP's Kristin McMahon details data cleansing best practices and explains why a good data cleanse needs continual communication, collaboration and oversight. Continue Reading
-
Tip
22 Mar 2019
Why data silos matter: Settling ownership of data issues
Data management is often still seen as an IT task, but that can lead to data silos. Find out why the business should be in charge of its data as part of a governance process. Continue Reading
-
News
21 Mar 2019
Machine learning approaches gain critical mass for data pros
Machine learning will bring change to analytics and data management, said data luminary Michael Stonebraker. Others agree managing such change will take special effort. Continue Reading
-
News
15 Mar 2019
Big data management practices gain attention amid risks
Many data professionals have yet to solidify traditional data management practices, but they have a new set of challenges to overcome to ensure data privacy and avoid misuse. Continue Reading
-
Feature
27 Feb 2019
How to navigate the challenges of the data modeling process
Data modeling and curation can help businesses more efficiently use data they've collected. There are challenges, however -- beginning with ensuring data quality. Continue Reading
-
Feature
27 Feb 2019
8 tips to improve the data curation process
A data curation and modeling strategy can ensure accuracy and enhance governance. Experts offer eight best practices for curating data. First, start at the source. Continue Reading
-
Tip
11 Jan 2019
Why organizations need a solid data governance strategy
The flood of data flowing into data warehouses, data lakes and other systems makes effective data governance a must for successful business analytics initiatives. Continue Reading
-
Feature
11 Dec 2018
Tips for creating curated data sets for self-service BI users
Data curation initiatives can help streamline BI processes by reducing the amount of time users spend locating and preparing data. Get four tips for preparing data sets. Continue Reading
-
Tip
16 Nov 2018
How data duplication in healthcare is diagnosed
Electronic health record systems have helped reduce duplicate patient data in hospitals -- but they haven't cured the problem. Find out how organizations are addressing the issue. Continue Reading
-
Tip
16 Oct 2018
11 features to look for in data quality management tools
As the need for quality data has increased, so have the capabilities of data quality tools. Learn how collaboration, data lineage and other features enable data quality. Continue Reading
-
Answer
27 Sep 2018
What is an enterprise data strategy?
Defining a data strategy can help focus an organization's data management initiatives -- but it isn't the same as data governance. Expert Anne Marie Smith explains why. Continue Reading
-
Answer
12 Sep 2018
What is the best structure for data governance programs?
The optimal approach to a data governance framework includes a program team, a data governance council and stewards. Expert Anne Marie Smith explains what role each group plays. Continue Reading
-
Answer
30 Aug 2018
5 to-dos for your GDPR compliance checklist
It's never too late to fine-tune your GDPR strategy. Expert Anne Marie Smith suggests a current state analysis of your PII protections, drafting a data privacy policy and more. Continue Reading
-
Tip
15 Aug 2018
Why data quality tools matter in today's organizations
Data is everywhere -- but is it good? Data quality tools could answer that question and help save time and money. So why don't more organizations use the technology? Continue Reading
-
Tip
09 Aug 2018
GDPR checklist items data managers might have overlooked
Now that the General Data Protection Regulation has been implemented, it's a good time for organizations to make sure they haven't missed any GDPR best practices. Continue Reading
-
Feature
31 Jul 2018
How to manage data and the GDPR right to be forgotten
The General Data Protection Regulation requires organizations to delete personally identifiable information if customers ask -- which can be a challenge. Experts offer four tips. Continue Reading
-
Tip
19 Jul 2018
Good data quality for machine learning is an analytics must
As companies add machine learning applications, they need to really understand -- and be able to improve -- their data. That's where data quality initiatives come in. Continue Reading
-
Tip
18 Jun 2018
Data integration techniques to help keep BI data consistent
Data integration processes that aren't managed properly can create inconsistent data in BI and analytics applications. Here are some steps to avoid that problem. Continue Reading
-
Tip
16 Apr 2018
GDPR rules can spur broader steps to protect sensitive data
Companies need to make compliance with GDPR's requirements on managing personal data a priority, but they should also work to implement wider data protection efforts in the future. Continue Reading
-
Feature
09 Apr 2018
Data integrity protection spurs greater security spending
As hacking, ransomware and malware attacks mount, companies place big data protection and integrity among the primary reasons for increased spending on security software. Continue Reading
-
E-Zine
09 Apr 2018
Security of big data fashions a whole new look with GDPR
As data managers scramble to protect their precious lakes of washed and unwashed data from the evils of hacking, malware, ransomware and botnets, there comes a privacy regulation of European Union origin that could change the way many U.S. companies protect their data from inside and outside forces. For some U.S. companies doing business in Europe and preparing for compliance, the EU's General Data Protection Regulation could be a well-intended enforcer of better data governance practices. But for businesses forced to change their old data protection habits, GDPR can be a four-letter word.
The April issue of Business Information opens with our editor's note and historical insights into the fundamental differences in attitude between the U.S. and Europe when it comes to protecting the data privacy of customers. As data breaches mount, however, the U.S. government -- willing or not -- may have to take steps beyond current regulations to ensure companies increase their security of big data.
Along those lines, our cover story examines some GDPR rules that can directly impact U.S. companies doing business overseas and, in the process, their data governance and ethics procedures. With the noted lack of maturity in data governance and security tools, we see in another feature how IT teams are addressing data security issues upfront in do-it-yourself ways when deploying big data systems.
Also in this issue, a GDPR compliance expert advises data managers on the best ways to prepare for a regulation that puts more control of information in the hands of users, the internet of things and edge computing could be compromising the security of big data and surveys show that companies place data protection among the top reasons for escalating their security spending.
Continue Reading -
Opinion
09 Apr 2018
U.S. data protection laws fall short in the age of big data
Data breaches and a history of data abuse led the EU to adopt GDPR, but it might take massive scale data security crises for the U.S. to legislate similar data protection laws. Continue Reading
-
Feature
09 Apr 2018
Data managers must hustle preparing for GDPR regulations
Data privacy expert and SAP consultant Stephen Lofthouse outlines a six-step process that data managers can follow to be compliant with GDPR regulations. Continue Reading
-
Feature
09 Apr 2018
Information architecture applied to big data streaming, AI
New technologies challenge data professionals, but taking a step back helps with hurdles. In this interview, consultant William McKnight takes a measured look at data streaming, GDPR and AI. Continue Reading
-
Feature
06 Apr 2018
Cloud workloads, data lakes challenge information architecture
Data management options are expanding; cloud workloads are an example. That means changing your approach to information architecture, says data management expert William McKnight. Continue Reading
-
Tip
30 Mar 2018
GDPR mandates push data quality improvement into IT spotlight
Data quality issues cost companies significant amounts of money in lost revenue and added expenses, and their impact will only get bigger when the EU starts enforcing its GDPR law. Continue Reading
-
Feature
22 Jan 2018
Infographic: The evolution of the chief data officer role
The CDO role, which has never been rigidly defined, is undergoing a face-lift as emerging technologies present new opportunities to enterprises. Continue Reading
-
Buyer's Guide
24 Oct 2016
How to determine which data governance tool best meets your needs
This buyer's guide provides insights to help you determine if your organization could benefit from data governance software and how to navigate the buying process. Continue Reading
-
Feature
28 Sep 2016
Examining the top data governance tools on the market
Expert John Ladley examines the leading data governance software products, comparing and contrasting their features to help you determine which tool will best meet your needs. Continue Reading
-
Feature
25 Aug 2016
How to narrow down your choices for buying a data governance tool
Evaluating and selecting a data governance tool depends on not only features and functionality, but also how you will use the tool to add value to your organization. Continue Reading
-
News
29 Jan 2016
Data quality metrics a governance must, managers say
Data governance managers who spoke during an online conference said that tracking business-oriented metrics on data quality improvement is a key to success in governance efforts. Continue Reading
-
Feature
13 Aug 2015
How to identify master data in a multi-domain MDM program
In an excerpt from their book on managing multi-domain master data management programs, Mark Allen and Dalton Cervo explain how to identify MDM domains and your master data. Continue Reading