Big data management
Big data management is the organization, administration and governance of large volumes of both structured and unstructured data.. See complete definition.
Top Stories
-
News
02 Feb 2022
Onehouse emerges with managed Apache Hudi data lake service
One of the original creators of the Hudi project at Uber has launched a new company set to bring a managed service to market to operationalize cloud data lakes. Continue Reading
-
News
27 Jan 2022
Confluent Kafka connects apps on new Dish 5G network
Streaming data is at the foundation of a new smart 5G service from Dish Network, where enterprises build applications that directly benefit and customize the network platform. Continue Reading
-
Feature
09 May 2019
Data modeling software tackles glut of new data sources
Data modeling platforms are starting to incorporate features to automate data-handling processes, but IT must still address entity resolution, data normalization and governance. Continue Reading
-
News
07 May 2019
ProvenDB brings blockchain applications to world of MongoDB
Blockchain is intriguing technology, but carries with it high system overhead. ProvenDB adds blockchain to MongoDB in an effort to gain acceptable performance. Continue Reading
-
News
30 Apr 2019
Snowflake CEO Bob Muglia talks cloud data warehouse evolution
In this Q&A, now-former Snowflake CEO Bob Muglia discusses the vendor's decision to embrace cloud data warehousing and how the industry is changing as more data moves to the cloud. Continue Reading
-
Feature
29 Apr 2019
Wayfair charts open source components course to growth
Teams at Wayfair mix new open source tools to power customer-facing apps. In such shops, tech leaders like Ben Clark must deftly maneuver an obstacle course of data components. Continue Reading
-
News
17 Apr 2019
Google takes a run at enterprise cloud data management
New Google Cloud boss Thomas Kurian is putting databases and data management at the forefront at Google. The vendor has forged key data deals, showing a more mature Google Cloud. Continue Reading
-
Feature
17 Apr 2019
4 factors to consider in a Hadoop distributions comparison
Examine the key characteristics necessary to evaluate in a Hadoop distribution comparison, focusing on enterprise features, subscription options and deployment models. Continue Reading
-
News
15 Apr 2019
Kafka at center of new event processing infrastructure
Events are as important as data in emerging applications underlying many e-commerce efforts. Streams of events tell a company what motivates customers to use online products. Continue Reading
-
News
04 Apr 2019
Tools manage performance for big data cloud applications
Tools such as Unravel and Pepperdata offer a way to measure performance of big data cloud applications, which may aid companies with on-premises configuration issues. Continue Reading
-
News
01 Apr 2019
Tools fill gaps in predictive modeling and machine learning
Machine learning platforms can bring the lone data scientist into the overall workflow. Updates to tooling also let that data scientist use familiar development interfaces. Continue Reading
-
Tip
29 Mar 2019
5 things to know about deploying big data systems in data containers
Planning for security and container APIs, and watching out for infrastructure sprawls are some issues to be aware of before deploying big data in containers. Continue Reading
-
News
25 Mar 2019
Facebook alumni forge own paths to big data analytics tools
Startups Interana and Rockset differ in their approaches to providing new query capabilities on fast-arriving big data. Both are led by technologists who started at Facebook. Continue Reading
-
News
20 Mar 2019
Aparavi update adds better data classification, more clouds
Aparavi updates Active Archive with customizable data classification and tagging capabilities for easier future retrieval and improved metadata search. Continue Reading
-
News
12 Mar 2019
Aerospike database garners Spark, Kafka connectors
Apache Kafka and Apache Spark connectors ease use of the Aerospike NoSQL data store in high-speed applications such as analytics that are becoming more broadly supported. Continue Reading
-
News
11 Mar 2019
Data catalog software takes on data lakes, privacy laws
Data catalogs form a hub for managing enterprise data. New products focus on machine learning and AI add-ons that help automate aspects of data governance. Continue Reading
-
Feature
25 Feb 2019
Explore Hadoop distributions to manage big data
Discover the uses of Hadoop distributions and the first steps in evaluating these products, as well as how the merger of rivals Cloudera and Hortonworks affects the market. Continue Reading
-
News
14 Feb 2019
Originators form group to boost Presto SQL query engine
The Presto engine arose as an alternative to Hive for big data queries. Now, the Presto Software Foundation has formed to promote the SQL query software's virtues. Continue Reading
-
News
01 Feb 2019
Open source cloud databases battle software 'strip mining'
Cloud giants like AWS have adopted open source databases, causing Confluent, MongoDB and others to guard their assets the best way they know how: licensing. Continue Reading
-
News
01 Feb 2019
Cloud data management, security top of mind for government
Federal government data officers grapple with cloud data management, weighing lower cost and efficiencies against security threats and vendor lock-in. Continue Reading
-
Feature
01 Feb 2019
Cloud data warehouse makes inroads as users spurn admin tasks
Overlooked in the run-up to Hadoop, data warehouses have found new life off premises. Cloud-based data warehouses find favor with teams that want to reduce warehouse administration. Continue Reading
-
Feature
23 Jan 2019
Advantages of graph databases: Easier data modeling, analytics
Graph databases are finding a place in analytics applications at organizations that need to be able to map and understand the connections in large and varied data sets. Continue Reading
-
Feature
16 Jan 2019
Southern Water’s centralised data team geared for silo busting
Southern Water has centralised its data specialists and overhauled its data management and business intelligence technology to support business decision-making at scale Continue Reading
-
News
15 Jan 2019
Cloudera and Hortonworks combo to push CDP, machine learning
Two wunderkinds of Hadoop have formalized their merger. Cloudera and Hortonworks say they will place special focus on AI as they chart the stand-alone vendor's future. Continue Reading
-
News
28 Dec 2018
Data management trends for 2019: Governance, DataOps, cloud
Better data governance, increased cloud use and wider DataOps adoption head the list of trends for data management teams to plan for in 2019, IT analysts say. Continue Reading
-
Podcast
19 Dec 2018
Open source support was central to 2018 data deals
Mergers and acquisitions unsettled the big data status quo in 2018. Open source support made these couplings a bit different than those of the past, Talking Data podcasters said. Continue Reading
-
Tip
17 Dec 2018
SQL Server 2019 improves Linux, container support
The SQL Server 2019 release includes new big data integration features, a collection of database engine enhancements and improved Linux and container support. Continue Reading
-
News
14 Dec 2018
Data companies vie to fill spots in AWS cloud data lineup
Third-party vendors that offer data platforms to AWS users tout hedges against cloud lock-in. But they must both compete and collaborate with the cloud leader. Continue Reading
-
Tip
20 Nov 2018
Trifacta data prep tool helps blend disparate data sources
Handling diverse data sources usually consumes precious developer time. That led healthcare CRM company SymphonyRM to hand the data prep task to business analysts. Continue Reading
-
News
12 Oct 2018
MarkLogic Data Hub Service aims to ease cloud use of NoSQL DBMS
MarkLogic rolled out a cloud-service version of its NoSQL database management system, a move designed to make the technology more cost-effective for cloud users. Continue Reading
-
Feature
11 Oct 2018
Cloud buoys data microservices -- for on-premises systems, too
Data in a microservices architecture is percolating anew. This news analysis looks at IBM Cloud Private for Data and other means to harmonize data in public and private locations. Continue Reading
-
News
04 Oct 2018
Cloudera-Hortonworks merger narrows Hadoop users' options
Hadoop users will have fewer choices as big data rivals Cloudera and Hortonworks unite. But the new company may be more competitive with AWS and Google. Continue Reading
-
Podcast
27 Sep 2018
Big data platform broadens place in analytics architecture
Big data platforms stumbled a bit getting out of the prototyping stage. But a view from the Strata Data Conference in New York sees broader use in the offing. Continue Reading
-
Opinion
21 Sep 2018
5 trends driving the big data evolution
The speedy evolution of big data technologies is connected to five trends, including practical applications of machine learning and cheap, abundantly available compute resources. Continue Reading
-
News
13 Sep 2018
Containers key for Hortonworks alliance on big data hybrid
Hortonworks is joining with Red Hat and IBM to work together on a hybrid big data architecture format that will run using containers both in the cloud and on premises. Continue Reading
-
News
16 Aug 2018
Chief data officer skills tested by AI tech blitz
A reporter's notebook from a recent MIT symposium provides insights on chief data officer needs, as the AI wave starts to hit CDOs and affects them more than most other tech trends. Continue Reading
-
News
08 Aug 2018
Confluent Platform 5.0 aims to mainstream Kafka streaming
Confluent Platform updates seek to bring data streaming with Apache Kafka to a wider audience. A new GUI and user-defined functions are part of the 5.0 release. Continue Reading
-
News
01 Aug 2018
Alluxio adds connectors for multi-cloud data migration
Updated Alluxio open source storage software homes in on portability across multiple big data object stores. Applications access Alluxio as mountable file storage. Continue Reading
-
News
19 Jul 2018
Focus, scope and spotting opportunity are key to role of CDO
Chief data officers and experts see the CDO role as changing to a more strategic orientation -- especially finding key opportunities in vast troves of data. Continue Reading
-
News
16 Jul 2018
Chief data officer role: Searching for consensus
The chief data officer role is about many things -- regulations, innovation, AI and more. Consultant Randy Bean discussed the matter ahead of an MIT symposium on the topic. Continue Reading
-
Feature
09 Jul 2018
eHarmony hooks up with Redis NoSQL database for hot storage
The Redis key-value store finds use in a system to match would-be romantic partners on dating site eHarmony, which employs a variety of NoSQL databases to make love click online. Continue Reading
-
Tip
06 Jul 2018
When to choose an S3 big data environment over HDFS storage
Selecting a storage service for big data in the cloud can be challenging. Expert David Loshin explains usage patterns that could lead organizations to Amazon Simple Storage Service. Continue Reading
-
Feature
03 Jul 2018
Labeled data brings machine learning applications to life
The types of data being collected for analytics use are increasing, but traditional structured data is a good match for machine learning. Gartner's Svetlana Sicular explains why. Continue Reading
-
Feature
02 Jul 2018
GPU cloud tools take complexity out of machine learning infrastructure
While talk of AI on GPUs is abuzz, actually building a machine learning infrastructure remains a dark art. A startup's PaaS is looking to automate parts of the process. Continue Reading
-
News
27 Jun 2018
MongoDB 4.0, Stitch aim to grow NoSQL apps in cloud, on-prem
NoSQL vendor MongoDB upgraded its database software with ACID support, while also releasing a serverless platform intended to simplify application development. Continue Reading
-
News
25 Jun 2018
Hadoop data lake architecture tests IT on data integration
Hortonworks users talk about building Hadoop data lakes to support new applications -- and the challenges their teams face on ingesting and refining data for end users. Continue Reading
-
News
18 Jun 2018
Hortonworks cloud options grow via Google, Microsoft, IBM
Hortonworks now supports Google Cloud Storage and has also broadened cloud deals with Microsoft and IBM, aiming to increase cloud uses of its big data platform. Continue Reading
-
Feature
07 Jun 2018
Google Cloud data lake fuels cloud payment processing flow
To create a cloud payment processing system, Global Payments first had to deploy a data lake in the Google Cloud. Getting quick user feedback was another early step. Continue Reading
-
Tip
05 Jun 2018
Why Spark DataFrame, lazy evaluation models outpace MapReduce
Learn how the Spark DataFrame execution plan works and why its lazy evaluation model helps the processing engine to avoid the performance issues inherent in Hadoop MapReduce. Continue Reading
-
Podcast
01 Jun 2018
Starburst finds new worlds to conquer with SQL query engine
Relational databases may have hit a wall of late, but the SQL query engine seems poised for wider growth. Starburst, a retro startup of sorts, is among those looking to take it wider still. Continue Reading
-
Feature
22 May 2018
AP uses data.world platform to spread data journalism
Data journalism reporters need tools that deliver quick context for stories on deadline. A data collaboration platform from data.world helps the Associated Press meet such needs. Continue Reading
-
News
18 Apr 2018
DataWorks 18: Hortonworks styles itself 3.0 with a ‘DataPlane’ service
Hortonworks has used its DataWorks Summit in Berlin to announce a data governance “studio” plug in to its DataPlane service Continue Reading
- 09 Apr 2018
-
Feature
09 Apr 2018
IT teams take big data security issues into their own hands
Data security needs to be addressed upfront in deployments of big data systems -- and users are likely to find they have to build some security capabilities themselves. Continue Reading
-
Feature
09 Apr 2018
Information architecture applied to big data streaming, AI
New technologies challenge data professionals, but taking a step back helps with hurdles. In this interview, consultant William McKnight takes a measured look at data streaming, GDPR and AI. Continue Reading
-
Feature
06 Apr 2018
Cloud workloads, data lakes challenge information architecture
Data management options are expanding; cloud workloads are an example. That means changing your approach to information architecture, says data management expert William McKnight. Continue Reading
-
Podcast
29 Mar 2018
Kubernetes container orchestration gets big data star turn
The new thing in big data is Kubernetes container orchestration. While it's still early, there are signs of activity, which are cited in this edition of the Talking Data podcast. Continue Reading
-
Feature
21 Mar 2018
Kubernetes gains momentum in big data implementation process
Big data vendors and users are looking to Kubernetes-managed containers to help accelerate deployments and enable more flexible use of computing resources. Continue Reading
-
Definition
21 Feb 2018
TensorFlow
TensorFlow is an open source framework developed by Google researchers to run machine learning, deep learning and other statistical and predictive analytics workloads. Continue Reading
-
News
16 Feb 2018
MongoDB 4.0 takes ACID transactions to multi-document level
MongoDB is taking a deeper step into SQL-style processing waters with a 4.0 update that brings increased support for ACID-compliant transactions to its NoSQL database. Continue Reading
-
Feature
06 Feb 2018
Data catalog software takes open data initiative to the streets
Data catalog software from Alation helped San Diego's chief data officer open up the city's data to residents, as well as city planners and other government workers. Continue Reading
-
Opinion
02 Feb 2018
Data managers should study up on GPU deep learning
As GPU deep learning becomes more common, data managers will have to navigate several new layers of complexity in their quest to build or buy suitable data infrastructure. Continue Reading
-
Tip
01 Feb 2018
Three ways to turn old files into Hadoop data sets in a data lake
Hadoop data lakes offer a new home for legacy data that still has analytical value. But there are different ways to convert the data for use in Hadoop depending on your analytics needs. Continue Reading
-
Podcast
23 Jan 2018
How AI and IoT will influence data management in 2018
AI and IoT will alter the data management landscape in 2018, according to analyst James Kobielus. AI will need regular updates, and DevOps will become more prevalent as a result. Continue Reading
-
News
03 Jan 2018
Apache Hadoop 3.0 goes GA, adds hooks for cloud and GPUs
Is this the post-Hadoop era? Not in the eyes of Hadoop 3.0 backers, who see the latest update to the big data framework succeeding in machine learning applications and cloud systems. Continue Reading
-
Feature
19 Oct 2017
NewSQL databases rise anew -- MemSQL, Spanner among contenders
The NewSQL database was almost hidden when Hadoop and NoSQL arose. Now, as more big data teams move toward production uses, MemSQL, Cloud Spanner and similar products may get a second look. Continue Reading
-
Feature
21 Aug 2017
Data management processes take on a new tenor in analyst's view
Changes in data management processes -- including self-service data preparation, data lakes and real-time analytics -- create a new landscape in companies, according to analyst Matt Aslett. Continue Reading
-
Feature
10 Aug 2017
Find your niche in big data: Consulting, software, platforms
Channel partners looking for a niche in big data may find opportunities in consulting, predictive analytics system deployment and software implementation. Continue Reading
-
News
24 Jul 2017
Chief data officer jobs call for nurturing data ethos in companies
In many organizations, chief data officer jobs centered on defense against risk are giving way to ones emphasizing innovation. To do so, CDOs must nurture a data culture, MIT panelists said. Continue Reading
-
Quiz
11 Jul 2017
Quiz: What's your IQ on data lake implementation and management?
A data lake can provide lots of analytics value to an organization. This quiz can tell you if you know what it takes to deploy and manage one; it also contains info to help boost your knowledge. Continue Reading
-
News
02 Jun 2017
Doug Cutting, ‘father’ of Hadoop, talks about big data tech evolution
In a Q&A interview, the “father” of Hadoop, Doug Cutting, talks about the cyber-security applications of the stack, as well as Hadoop’s evolution Continue Reading
-
Feature
13 Mar 2017
Silicon Valley startups aim to make big data capture and prep slicker
A group of California-based startup and early-stage data analytics and management companies are bidding to make big data, including sensor data, more tractable for analysis Continue Reading
-
News
15 Sep 2016
HPE sells Vertica analytics, thanks to the growth of open source software
HPE is paring down its software holdings, including analytical database software in the Vertica line and other big data tools. A sale to Micro Focus is due to close next year, leaving users in some limbo for now. Continue Reading
-
Tip
19 Jul 2016
Salesforce rallies in retail with purchase of Demandware e-commerce
Now that Salesforce has purchased Demandware, it's positioned to compete with vendors like Oracle and even Amazon for market share. Continue Reading
-
Feature
26 Apr 2016
Inside the MapR Hadoop distribution for managing big data
The MapR Hadoop distribution replaces HDFS with its proprietary file system, MapR-FS, which is designed to provide more efficient management of data, reliability and ease of use. Continue Reading
-
Feature
22 Apr 2016
CW@50: Data management – Five decades of prospecting for business value
Data management and business intelligence have been at the heart of business value creation for decades. Read about how Computer Weekly has tracked their promise and tribulations Continue Reading
-
Feature
29 Oct 2015
Three ways to build a big data system
In a book excerpt, author Dale Neef outlines and compares different approaches organizations can take when trying to bring a big data system into their IT environments. Continue Reading
-
Podcast
25 Aug 2015
The latest trends in the Hadoop project
In this podcast, Tim Hall of Hortonworks discusses the Apache Hadoop project and how it has changed since its inception. Continue Reading
-
Feature
14 Aug 2015
Types of NoSQL databases and key criteria for choosing them
In a book excerpt, consultant Dan Sullivan offers insights into how to select the right type of NoSQL database for the right application in your organization. Continue Reading
-
Feature
20 Jul 2015
Business cases emerge from growing pains of Hadoop 1.0
The hype around the early generation of Hadoop is giving way to the reality of business programmes based on Hadoop 2 and Apache Spark Continue Reading
-
Feature
11 Jun 2015
Which in-memory DBMS best fits your company's needs?
Database expert Craig S. Mullins examines some top in-memory DBMS software offerings to help you determine which is the best fit for your organization. Continue Reading
-
E-Zine
07 May 2015
Sort through types of database management systems before you buy
The database management system lies at the center of any organization's business activities. It houses the data that pumps life into operations and analytics applications and ensures that it's managed and manipulated securely. But the types of database management systems, or DBMSes, are numerous -- and organizations starting the evaluation process need to first examine the data they need to process, says data management consultant Craig S. Mullins in this special edition of Business Information. Is it traditional, structured data? Or is it a vast pool full of audio and video streams, social media data and other unstructured information?
First, Mullins surveys the database product landscape, detailing the advantages of the types of database management systems on the market, breaking them into three main categories: relational database management systems, NoSQL databases and in-memory databases. Next Mullins takes a close look at relational systems. Any organization shopping for a database system should begin here, he says, because relational databases are good for a wide range of uses -- traditional data management and analytical applications, batch processing and more. Finally, Mullins explores the relatively new world of NoSQL database systems, known for their able handling of big data. NoSQL has its own types of database management systems, falling into four categories: key-value, document, column and graph. Each is suitable for particular uses, so companies may want to consider mixing and matching, depending on their data needs. Continue Reading -
Opinion
01 May 2015
With database management tools, never settle
They're the heart of it all, pumping data to applications and business processes. But the sheer number of products on the market can boggle any mind. Continue Reading
-
Feature
30 Apr 2015
Which relational DBMS is best for your company?
Database expert Craig S. Mullins examines the top relational database management system software to determine which best fits your organization. Continue Reading
-
Feature
24 Apr 2015
Apache Spark speeds up big data decision-making
Spark, the open-source cluster computing framework from Apache, promises to complement Hadoop batch processing Continue Reading
-
Feature
17 Apr 2015
How to determine if an in-memory DBMS is right for your company
Database expert Craig S. Mullins examines the pros and cons of the in-memory database management system, and the criteria you should consider during the request-for-proposal and evaluation period. Continue Reading
-
Feature
09 Jan 2015
Big data and analytics advice from an operations pro
You don't need to employ special IT personnel for big data analytics. One company uses big data for developers to modify ads and find and adjust pain-points in a game application. Continue Reading
-
News
04 Jun 2014
Apache Spark goes 1.0, looks to improve on MapReduce performance
The Apache Software Foundation released Version 1.0 of Spark, a processing framework aimed at outpacing MapReduce for machine learning and other uses. Continue Reading
-
Opinion
30 May 2014
Anxieties about big data technology trends are valid
Criticism of big data technology trends may go too far, but proponents shouldn't dismiss the critiques. Continue Reading
-
Answer
29 May 2014
Seven best practices to boost big data governance efforts
Governing big data is much like governing other data, though businesses must adjust governance processes to accommodate larger, more varied data sets. Continue Reading
-
News
08 May 2014
Big data charge could turn data management profession on its head
Discussed at DAMA's EDW 2014 were more flexible approaches to the data management profession. That's in response to growing mountains of diverse data. Continue Reading
-
Opinion
28 Apr 2014
Big data cloud is changing familiar database tech tune
New cloud database applications that increasingly fall into the "big-data-intensive" category are shaking up the usual build-or-buy soliloquy. Continue Reading
-
Feature
17 Apr 2014
Data collection practices spark debate on big data ethics, privacy
The increasing emphasis on collecting and analyzing big data is driving a debate on whether tighter privacy rules are needed to protect customers. Continue Reading
-
Feature
14 Apr 2014
Supermarket co-op stocks up on big data platform to help spur sales
Co-operative Allegiance Retail Services is deploying a cloud-based big data stack to replace a homegrown system that fell short on analytics power. Continue Reading
-
Feature
31 Mar 2014
Selecting the right SQL-on-Hadoop engine to access big data
Rick van der Lans explains why it's important to evaluate the differences in the technologies that make it possible to access Hadoop data using SQL. Continue Reading
-
Feature
27 Feb 2014
Hadoop 2 opens new vistas beyond current UK data management practice
Hadoop 2 enables a greater range of data-crunching tasks, beyond Hadoop’s previous scope of batch processing, directly within the stack itself Continue Reading
-
News
09 Sep 2013
Survey: Execs buying into big data in a big way
TechTarget's Wayne Eckerson discusses survey results showing that two-thirds of respondents expect to spend more than $1 million on big data programs this year. Continue Reading
-
Feature
29 Aug 2013
Big data adds new complications for data stewards
In order to accommodate big data and the needs of data scientists, data stewards will need to adjust to a more short-term view of data stewardship. Continue Reading
-
Tip
22 Apr 2013
Real-time querying in Hadoop? Not so fast
Hadoop vendors are adding real-time query engines to the open source software. But the jury is out on that effort, writes Wayne Eckerson. Continue Reading
-
Feature
04 Feb 2013
Without analytics, it's 'big data, big schmada'
Data -- including big data -- has no value by itself, says TechTarget's Wayne Eckerson. It's what you do with data to gain business insights that counts. Continue Reading
-
Feature
15 Nov 2012
Hadoop World: Just like data warehousing's early days
The buzz-filled Hadoop World 2012 conference reminded TechTarget's Wayne Eckerson of a data warehousing event he attended in 1997. Still to be seen: Can Hadoop live up all to the hype? Continue Reading
-
Feature
05 Nov 2012
Unilever enterprise data warehouse locked to business programmes
Unilever is building a global enterprise data warehouse (EDW) locked into the critical paths of high-profile business programmes. Continue Reading