Manage Learn to apply best practices and optimize your operations.

McKnight: Building a big data ecosystem not a cookie-cutter concept

In the big data era, enterprise architectures are no longer nice, neat and replicable from company to company, according to business intelligence and data management consultant William McKnight. In fact, he uses the concept of a "no-reference architecture" in discussing the technologies that could be incorporated into a big data ecosystem in an organization. "Every company is different," McKnight said in a video interview with SearchDataManagement. "Gone are the days when a vendor or a consultant could walk into a shop with a laminated sheet of paper and say, 'This is what everybody needs to do.'"

And in most cases, McKnight added, a big data environment requires a variety of different technology platforms. "Everybody has a dirty sheet of paper right now with all sorts of lines crisscrossing about data integration," he said in the interview, recorded at the 2014 TDWI World Conference and BI Executive Summit in Las Vegas. "The idea is just to keep moving it forward, though; keep moving it forward into a modern architecture that stores all data and serves it up to the user community."

McKnight is president of McKnight Consulting Group and author of Information Management: Strategies for Gaining a Competitive Advantage with Data, published in December 2013. He said the traditional data warehouse still has a role to play in supporting "your basic type of reporting needs." But it can't meet all the demands of a modern IT infrastructure -- especially a big data architecture. Columnar databases, in-memory processing systems, and other technologies are also called for, McKnight said.

And then there's Hadoop. McKnight said the open source distributed processing framework is starting to find its way into most large companies -- not necessarily for production uses at this point, but for prototyping and proofs of concept. Hadoop also can't go it alone, though. "It has to work within the ecosystem," he said. "It's not going to do away with the data warehouse or relational databases in general. It's additive to an environment."

Watch the six-minute video to hear more from McKnight on building a big data ecosystem, including the importance of being prepared to handle a big data analytics program. For example, he said he encourages his clients to "really improve their data science so they're able to take advantage of the information, because the technology is there to store all of the data."

Text by Craig Stedman, executive editor. Email him at [email protected], and follow us on Twitter: @sDataManagement.

View All Videos