A new era of continuous streaming IoT analytics

The dawn of a new analytics era

Years ago, self-service BI tools put the power to analyze data into the hands of domain experts. By reducing the dependency on IT, these tools ushered in a new era of insight-driven business. Suddenly, if you could use a computer, you could use analytics tools such as Tableau, Qlikview, or Spotfire on your own without waiting weeks or months for IT.

Self-service visual exploration fueled dramatic growth in the BI market, thanks to the productivity gains from making opaque business processes, transactions and data fully transparent to domain experts and process stakeholders. For the first time, they could finally see what had been previously hidden.

A new, similar era of disruption is now upon us, because recent innovations in self-service streaming analytics empower business users to see IoT data like they have never seen before. These technologies are streaming BI and data science in real time.

Streaming BI

Streaming BI is like an algorithmic canary in the IoT data coalmine. It provides business users continuous, live insight into what’s happening now by connecting directly to streaming IoT data.

From a business analyst point of view, streaming BI works like a conventional BI tool, but instead of connecting it to historical data, you connect it directly to information in motion: IoT data flowing through messaging systems like Kafka, MQTT, OSI PI or OPC UA, an increasingly popular IoT protocol on Microsoft’s Azure platform.

The magic happens after you create a BI dashboard. Streaming BI remembers the queries needed for each BI visualization and continuously monitors data streams for changes in the query result. The moment the result changes, visualizations update.

Users can create alerts that use rules or data science algorithms to monitor IoT sensor data. When the incoming data shows changes in defined alert conditions, the dashboard changes and alerts are delivered according to rules the business user creates.

For example, you can ask an algorithm to display all trains on a map, highlight in red trains that are predicted to be more than 1 minute late, send me a text message if that prediction becomes greater than 5 minutes late. This visualization constantly updates without re-asking the question. Even if you’re not sitting at your desk, you’ll be notified when something happens that makes or increases the chances a train will be unusually late.

This form of BI completely changes the interaction model between the analyst and data. Insights are live algorithms pushed to the analyst directly. Analysts can just set it and forget it. The public can be alerted that a train is arriving late while they are still driving to the station. Better yet, the problem can be fixed before it happens, and passengers will never know there might have been a problem in the first place.

Now, imagine instead of a train running late, sales are not materializing or a storm is coming that might affect safety. Anticipating such outcomes before they materialize and in real time while you can still do something about it is critical.

Streaming data science powers streaming BI

The second recent innovation in IoT analytics is streaming data science.

The AI field of adaptive learning is the AI equivalent of how humans learn: instead of training models only on what has already happened, adaptive learning can train models on changing data streams by injecting Python, Tensorflow or Java data science algorithms into streaming data for continuous evaluation.

Streaming data science presents new opportunities for adaptive learning. For example, in high-tech manufacturing, a nearly infinite number of different failure modes can occur. To avoid such failures, machine learning models applied to IoT data can help identify patterns associated with quality problems as they emerge and as quickly as possible. When never before seen root causes — such as machines or manufacturing inputs — begin to affect product quality, staff can respond more quickly.

Adaptive learning with streaming data means continuous learning and calibration of models based on the newest data. Sometimes applying specialized algorithms to streaming data can simultaneously improve the prediction models and make the best predictions at the same time.

When you combine streaming data science and streaming BI, business analysts can gain continuous algorithmic insights. This helps democratize data science models by helping humans manage automated algorithmic insights.

Self-service streaming BI and streaming data science at work

These innovations open an entirely new market for BI: digital operations. Until now, BI was useless for digital operations because there’s no value in understanding how you should have acted yesterday to avoid problems or seize opportunities.

For example, smart city transportation analysts can now understand and act on congestion in real time by evaluating sensor data from vehicles, satellite imagery and weather forecast data with streaming BI. Analysts can receive notifications when data science models predict that problems are about to occur and decide how to reroute vehicles, notify drivers or alert the public to avoid problems before they happen.

Other examples of streaming BI and data science at work include:

• Shipping operations can be alerted when a model predicts that a cargo ship will arrive earlier than expected and doesn’t have a place to dock. In shipping, wasted time is wasted money, and shipping operators can use real-time insights to reoptimize port operations and eliminate delays.
• Insurance adjusters can continuously monitor weather patterns and models that predict areas where claims will be prevalent so that they can proactively alert customers and schedule adjusters to reduce financial loss and improve customer engagement.
• High-tech manufacturing operations can monitor sensor readings from equipment, apply adaptive learning models and predict and fix problems before they impact production or quality.
• Online marketers and web user experience designers can evaluate how specific offers are working, change layout and evaluate goal attainment on a commerce site.

Until recently, continuous insights like these were only possible through laborious custom programming. Now, self-service streaming BI and streaming data science puts real-time power in the hands of analysts.

Stop looking at the past and start looking toward the future

Today, nearly every business is stuck looking only at the past. Thanks to the plummeting cost of IoT sensors and innovations in streaming analytics, continuous insights can now be made available to any business user. It’s as easy as opening a spreadsheet.

The agility gained from self-service continuous insight allows subject matter experts to understand what’s happening now, predict problems before they occur and decide how to act while the game is still on.

A decade ago, self-service BI disrupted the analytics market; now, self-service streaming BI promises to democratize continuous, real-time insight. Old-school BI will always have its place, but with ubiquitous IoT data, business users can stop looking at what could have been, get in the digital game, act and win.

All IoT Agenda network contributors are responsible for the content and accuracy of their posts. Opinions are of the writers and do not necessarily convey the thoughts of IoT Agenda.