6 top business benefits of real-time data analytics

1. Make decisions at the speed of your business

Much of what we do in the world of data management, BI and analytics -- even with the latest data science and advanced analytics tools -- still falls under the rubric of that old term decision support. It's a good descriptor: the software supports you in making better decisions.

As I suggested above, better decisions do need to be faster decisions as the pace of business increases. When even modest businesses are trading globally online, it's not good enough to wait for an overnight data management process to reconcile the data warehouse in order to deliver standardized reports. Business teams working remotely or internationally need the best information continuously, more often delivered in a dashboard with immediate insights.

So, the first -- and for many the primary -- benefit of real-time data in the enterprise is simply being able to support decisions whenever and wherever they need to be made.

2. Increase business agility and optimization

It's easy to fall into the trap of thinking that faster decision-making means a more agile business. But in fact, business agility isn't only about decisions -- agility also encompasses your strategic and tactical business goals. Choosing between options quickly doesn't help when you should be making choices of a different kind.

One approach to business agility that has proven very successful in several industry sectors is the creation of small, well-informed, tightly focused teams called squads. For example, retailers may use squads to focus on produce, home goods or other specific merchandise categories, empowering them to quickly and directly make decisions that previously might have involved considerable management review. Manufacturers may have squads that focus on maintenance or safety.

By giving squads a mandate to act quickly so close to your operations, you enable faster and -- hopefully -- smarter responses to a changing business environment. However, this approach can only be effective if a squad has the needed data, which must continuously be kept up to date to match the team's urgency. That's a great use case for real-time data.

3. Quickly detect and address operational issues

Squads started in the tech industry but have proven popular in sectors from retail to telecom and even, recently, healthcare -- all businesses facing rapidly changing markets and cost pressures, always with an eye to their profit margins.

You don't necessarily need a squad to improve business operations with real-time data. Using data from IoT sensors or video feeds to monitor production lines for stoppages and backlogs and run predictive maintenance applications has been popular for several years now. It is a classic example of real-time operational improvement in action and has proven very effective at reducing downtime in manufacturing plants.

Similar approaches can be applied in quite different scenarios. For example, real-time reports of traffic conditions and weather forecasts can help logistics companies route delivery trucks more effectively. If those trucks are refrigerated, onboard temperature sensors can also monitor for issues requiring immediate attention or rerouting and send real-time alerts.

Companies also use real-time analytics to monitor the balance of incoming orders and product or parts availability so they can move quickly to increase supplies that are running short, and to detect the need for short-term contract labor if production, packaging or shipping is falling behind target.

4. Identify and act on short-term market changes

Some industries are clearly sensitive to rapid market fluctuations -- stock trading is an obvious example. Naturally, real-time data is essential to business survival in such cases.

Other industries, such as airlines and hotel chains, manage prices and availability based on current events, weather, oil prices and other factors that can change rapidly. Today, with so much of the retail experience taking place online, retailers too have to respond quickly to changing demand, costs and customer trends.

Real-time data is invaluable to all these scenarios. It's possible to move more slowly if your inventories and margins allow you to take a deep breath and ride through some disruptions. However, few businesses have such luxury these days. Instead, we have had to become much smarter about using data to enable faster and more efficient monitoring of our markets.

5. Personalize the customer experience for online marketing

Online retail is a prime example of how real-time data enables new and more effective customer experiences. In the old days of the brick-and-mortar store, attentive staff would recognize, greet and guide the regular and best customers. Today, you're more likely to be recognized by a bot that has access to real-time data from your online behavior. And while the bot may not greet you, it certainly will ensure that the homepage, special offers, recommendations and, in some cases, even the color scheme reflect what it has learned about you over numerous sessions.

Some people think this attentiveness and automated customization can be "creepy" and a little too focused for comfort. But the truth is, numerous organizations use real-time technology to personalize their websites and online advertising for individual customers, without people even realizing that it is working behind the scenes to serve up what they think of as the normal customer experience.

6. Improve customer service with up-to-date information

Whether you're calling customer service at your utility company, cable provider, mobile network operator or airline, your experience should be better now than it was just a few years ago. Why? Because all of these industries, and many others, have invested heavily in real-time data integration for their call center operations.

When call center agents look up someone's customer records, they should be able to see information on the local outage, faulty equipment, unusually high bill, canceled flight or other issue that prompted the call, right then as they're talking with the customer. This kind of insight driven by real-time applications is standard practice now.

Collecting and managing real-time data

It's difficult to give a concise description of all the processes involved in real-time data management. However, it's important to call out two somewhat different approaches to collecting and managing data: micro-batching and streaming.

Most traditional data repositories, such as data warehouses and operational databases, are loaded with data in batches and respond to analytical queries with a data set that itself is a kind of batch. When processing a batch, there is a beginning and an end. Extract, transform and load is the most common form of data integration; I've seen batch ETL processes that ran for 12 hours, loading a data warehouse with millions of records. That's a big batch, and it was always a relief when it ended successfully.

If we want near-real-time data collection -- on some type of performance data, for example -- we can process micro-batches, sometimes only one record at a time. If the data management environment can handle it, these micro-batches can be processed into and out of the analytics system with great rapidity, approximating the real-time data at the source. Nevertheless, micro-batches still have a beginning and an end and should have some of the advantages of traditional data integration techniques, such as robust transaction handling in the event of a failure.

Like a flowing river, streaming data has no clear beginning or end. The streaming approach is particularly popular when collecting real-time data from sensors, such as the ones in devices connected to the IoT, but streams can also come from transaction logs, activity logs and other sources.

For simple types of analytics, such as reporting on current conditions, monitoring for exceptions and outages or optimizing business processes in response to real-time activity, streaming may be a great option. A good streaming system is capable of handling very large volumes of data, also making it suitable for data science applications.

However, if it's important to integrate real-time data from one source system with transaction data from another, micro-batching is likely a better solution. For example, handling airline tickets and re-bookings in the event of flight delays involves high volumes of rapidly changing data, but the airline needs to ensure that each change in the system is committed safely with a transaction. Micro-batching works for such scenarios.

Challenges of real-time data analytics

There are some real advantages and benefits to real-time data analytics for many businesses. But you need to be aware of some potential pitfalls. Even though most real-time data is now processed and stored in the cloud, the sheer scale that's often involved requires special planning for data storage. Indeed, much of big data, which can include both structured and unstructured data, is generated by sources that drive real-time analytics, such as web traffic logs and manufacturing equipment.

You also need to carefully consider your data archiving strategy if the decisions based on the data need to be audited or reviewed for governance and compliance. And you need to plan operationally, tactically and strategically for problems such as system outages, late-arriving data or other real-time processing problems.

Nevertheless, with effective real-time technologies now available from many leading analytics vendors, the real-time stream is rapidly becoming mainstream.