Browse Definitions :
business process reengineering (BPR) How to develop a business process in 8 steps

complex event processing (CEP)

What is complex event processing (CEP)?

Complex event processing (CEP) is the use of technology to predict high-level events likely to result from specific sets of low-level factors. By identifying and analyzing cause-and-effect relationships among events in real time, CEP allows personnel to take effective actions in response to specific opportunities or threats. For example, a team might create an event that aggregates several low-level risk indicators, such as change of address, new spending habits and new types of purchases to elevate fraud risks.

The concept and techniques of CEP were developed in the 1990s. The approach gained popularity in the heyday of service-oriented architectures and enterprise service busses, architectural styles which have since been eclipsed by cloud-based architectures. Likewise, CEP has been somewhat overshadowed by event stream processing and streaming analytics, two approaches for processing and analyzing streams of data that implement some, but not all, of the core ideas of CEP.

Despite the fact that different terminology is often used today for processing and analyzing real-time events, experts believe that CEP remains relevant for improving enterprise architectures, and, in particular, for its application in the discipline of business process management, which aims to improve business processes end to end.

Donncha Carroll, a partner in the revenue growth practice of Axiom Consulting Partners, said that business process modeling software (BPMS) tools have started to add new functionality that lets users handle CEP, making it easier to connect to multiple sources of enterprise data. This new CEP functionality in BPMS tools allows business process managers and business owners to identify events or developments that trigger critical notifications or actions that are generally not possible to identify with enterprise software platforms serving a broader set of needs, such as CRM and ERP tools.

A good example of a CEP use case of this type is an enterprise software vendor with a large, important customer coming up for renewal. The customer is experiencing a service disruption that no one but the customer service department is aware of -- putting the renewal at risk. CEP can put both pieces of information together -- up for renewal and service disruption -- prompting the system to trigger a notification and launch a set of activities that prioritize account servicing for this customer.


History of CEP

The history of CEP begins with work done by Professor David Luckman at Stanford University.

In the 1990s, Luckham was working on distributed programming languages that could run in parallel and communicate by means of events.

"It became clear that you needed to abstract low-level events into higher-level events and do that two or three times to create accurate simulations," said Luckham, now a professor emeritus at Stanford, in an interview with TechTarget.

For example, Intel came to Luckham's team when they were trying to figure out why the adder on a new chip did not work correctly. Initially, they thought the simulation library was deficient, but it turned out the analysis of the results lacked the ability to make sense of the raw data streams.

"In those days, simulation outputs were eyeballed by humans as streams of ones and zeros, and it was easy to miss something at that level," Luckham said. He invented the term complex events to characterize higher-level events correlated from a series of lower-level events.

Luckham's team outlined three core principles of CEP: synchronization of the timing of events, event hierarchies and causation.

  • Synchronization relates to the observation that the timings of events often need to be calibrated owing to differences in processing time, network routes and latency.
  • Event hierarchies relate to the development of models for expressing lower-level events, such as how clicks can translate into higher-level events like user journeys. Researchers today are developing tools for creating synthetic sensors that detect behavioral events by fusing data from various sensors.
  • Causation provides a framework for connecting the dots between cause-and-effect relationships buried within a stream of events.

Luckham observed that most tools have focused on synchronization. However, more work will be required to weave the science behind event hierarchies and causation into modern tools to take advantage of the original promise of CEP.

Meanwhile, CEP's techniques for identifying, analyzing and processing data in real time have become fundamental to many business projects.

"Today, many businesses plan strategic initiatives under titles such as 'Business Analytics' and 'Optimization.' Although they may not know it, complex event processing is usually a cornerstone of such initiatives," Luckham said.

Benefits of complex event processing

CEP's ability to detect complex patterns in multiple sources of data provides many benefits, including the following:

  • makes it easier to understand the relationship between events;
  • helps connect individual events into more complex chains;
  • simplifies the development and tuning of business logic;
  • can be embedded into fraud detections, logistics and IoT applications; and
  • helps build more accurate simulations, models and predictive analytics.

Use cases for complex event processing

The ways in which organizations use CEP today include the following:

  • improve stock market trading algorithms;
  • develop more responsive real-time marketing;
  • create more adaptable and accurate predictive maintenance;
  • fraud detection;
  • develop more nuanced IoT; and
  • enable more resilient and flexible supply chains and logistics.

Difference between complex event processing and event stream processing

With the growth of cloud architectures, new terms like event stream processing (ESP), real-time stream processing and streaming architectures are starting to replace the use of the term CEP in many technical discussions and product marketing.

Roy Schulte, distinguished vice president analyst at Gartner, has been covering CEP almost from its inception. Complex event processing as a mathematical concept is still valid, he said, but the term itself is not used that often. He hears it used in reference to temporal pattern detection -- finding new occurrences of a pattern in streaming data -- and to stream data integration, that is, the loading of streaming data into databases (streaming extract, transform and load).

The disuse of the term is due in part to how complex events are defined. For instance, events based on arithmetic aggregation operators (e.g., count the number of tweets in the last 10 minutes on the topic of COVID or other count, sum, average calculations) are often not called complex events. They're relatively simple to compute, so the software need not be as sophisticated as that used for pattern detection in CEP. Simple forms of stream analytics are now available in common BI tools like Tibco Spotfire, Tableau and Microsoft Power BI.

"The technology of CEP is generally called stream analytics now, or streaming analytics, by some," he said. Streaming analytics is widely practiced because organizations need to extract near-real-time insights from the vast amount of streaming data that flows through their networks (and internet connections). This enables situational awareness.

Sophisticated forms of stream analytics, such as pattern detection, are implemented in "event stream processing platforms" like Apache Flink, IBM Streams, Kafka Streams, Microsoft Azure Stream Analytics, Spark Streaming and Tibco StreamBase. "Such ESP platforms are often embedded, unseen, within SaaS, packaged applications or other tools so the buyer may not realize that they are there," Schulte said.  

The stream processing characteristic of CEP is also sometimes manually coded into SaaS, packaged applications or other tools, so the product implements the same math as CEP without using a CEP/ESP platform.

Still, most of the early CEP tools are still offered -- generally under the banner of stream analytics -- and remain viable. However, newer event stream processing platforms such as aforementioned Microsoft Azure Stream Analytics and open source ESP platforms like Flink, Spark Streaming and Kafka Streams have taken over the bulk of new applications.

Big organizations already use multiple stream analytics technologies, whether they are aware of this functionality or not. That's because these technologies are now widely embedded in SaaS applications or in packages for important activities such as transportation operations monitoring, network management or fraud detection. In other cases, they are embedded in BI tools or closed source or open source ESP platforms. "For the same reason that big organizations have a dozen or more DBMSes, they have multiple stream analytics products," Schulte said.

Most of these applications that employ event-driven architecture, however, don't use stream analytics or CEP. Instead, they use simple one-event-at-a-time events processing. In this sense, stream analytics and stream data integration are subsets of event-driven architecture applications.

This was last updated in April 2022

Next Steps

Continue Reading About complex event processing (CEP)

  • cloud-native network function (CNF)

    A cloud-native network function (CNF) is a service that performs network duties in software, as opposed to purpose-built hardware.

  • microsegmentation

    Microsegmentation is a security technique that splits a network into definable zones and uses policies to dictate how data and ...

  • Wi-Fi 6E

    Wi-Fi 6E is one variant of the 802.11ax standard.

  • MICR (magnetic ink character recognition)

    MICR (magnetic ink character recognition) is a technology invented in the 1950s that's used to verify the legitimacy or ...

  • What is cybersecurity?

    Cybersecurity is the protection of internet-connected systems such as hardware, software and data from cyberthreats.

  • Android System WebView

    Android System WebView is a system component for the Android operating system (OS) that allows Android apps to display web ...

  • privacy compliance

    Privacy compliance is a company's accordance with established personal information protection guidelines, specifications or ...

  • contingent workforce

    A contingent workforce is a labor pool whose members are hired by an organization on an on-demand basis.

  • product development (new product development -- NPD)

    Product development, also called new product management, is a series of steps that includes the conceptualization, design, ...

  • talent acquisition

    Talent acquisition is the strategic process employers use to analyze their long-term talent needs in the context of business ...

  • employee retention

    Employee retention is the organizational goal of keeping productive and talented workers and reducing turnover by fostering a ...

  • hybrid work model

    A hybrid work model is a workforce structure that includes employees who work remotely and those who work on site, in a company's...

  • Salesforce Trailhead

    Salesforce Trailhead is a series of online tutorials that coach beginner and intermediate developers who need to learn how to ...

  • Salesforce

    Salesforce, Inc. is a cloud computing and social enterprise software-as-a-service (SaaS) provider based in San Francisco.

  • data clean room

    A data clean room is a technology service that helps content platforms keep first person user data private when interacting with ...