7 steps to a successful data lake implementation

Do what needs doing with Hadoop data

The good news, however, is the challenges are easily overcome. Here are seven steps to address and avoid them:

1. Create a taxonomy of data classifications. Organizing data objects in a data lake depends on how they're classified. Identify the key dimensions of the data as part of your classifications, such as data type, content, usage scenarios, groups of possible users and data sensitivity. The latter relates to protecting both personal and corporate data -- such as personally identifiable information on customers in the first case and intellectual property in the second.
2. Design a proper data architecture. Apply the defined classification taxonomy to direct how the data is organized in your Hadoop environment. The resulting plan should include things like file hierarchy structures for data storage, file and folder naming conventions, access methods and controls for different data sets, and mechanisms for guiding data distribution.
3. Employ data profiling tools. In many cases, the absence of knowledge about all the data going into a data lake can be partially alleviated by analyzing its content. Data profiling tools can help by gathering information about what's in data objects, thereby providing insight for classifying them. Profiling data as part of a data lake implementation also aids in identifying data quality issues that should be assessed for possible fixes to make sure data scientists and other analysts are working with accurate information.
4. Standardize the data access process. Difficulties in effectively using data sets stored in a Hadoop data lake often stem from the use of a variety of data access methods, many undocumented, by different analytics teams. Instead, instituting a common and straightforward API can simplify data access and ultimately allow more users to take advantage of the data.
5. Develop a searchable data catalog. A more insidious obstacle to effective data access and usage is prospective users being unaware of what's in a data lake and where different data sets are located in the Hadoop environment, in addition to information about data lineage, quality and currency. A collaborative data catalog allows these -- and other -- details about each data asset to be documented. For example, it captures structural and semantic metadata, provenance and lineage records, info on access privileges and more. A data catalog also provides a forum for groups of users to share experiences, issues and advice on working with the data.
6. Implement sufficient data protections. Aside from the conventional aspects of IT security, such as network-perimeter defenses and role-based access controls, utilize other methods to prevent the exposure of sensitive information contained in a data lake. That includes mechanisms like data encryption and data masking, along with automated monitoring to generate alerts about unauthorized data access or transfers.
7. Raise data awareness internally. Finally, make sure that the users of your data lake are aware of the need to actively manage and govern the data assets it contains. Train them on how to use the data catalog to find available data sets and how to configure analytics applications to access the data they need. At the same time, impress upon them the importance of proper data usage and strong data quality.

To meet the ultimate objective of making a data lake accessible and usable, it's crucial to have a well-designed plan for dealing with the data prior to migrating it into your Hadoop environment or cloud-based big data architecture. Taking the steps outlined here will help streamline the data lake implementation process. More important, the right combination of planning, organization and governance will help maximize your organization's investment in a data lake and reduce the risk of a failed deployment.