Articles

What Does Complex Event Processing Do for BPM?

Discussions of event-driven architecture (EDA) and complex event processing (CEP) are appearing more frequently in articles and analyst discussions about business process management (BPM) and business activity monitoring (BAM). So what unique, valuable capabilities can these concepts and technologies introduce into our solution toolkits? And how can these capabilities improve our BPM solutions? This article explores those two central questions and provides a foundation for understanding whether and how you should consider using CEP.

What Is Complex Event Processing?

CEP technology is a highly responsive and adaptable mechanism for identifying business scenarios or events that require immediate attention. Combined with existing Business Process Execution Language (BPEL)-based BPM tools, it provides a comprehensive, agile foundation for business process optimization, ensuring that business processes are efficient and flexible.

A typical organization generates millions of events every day. These events may be system events created by our systems and applications or transaction events created by customers, suppliers, employees, or other stakeholders. Given this wide range of potential sources of raw events, they are usually represented in many different forms and formats.

In many cases, these events are explicit and easily recognized and can therefore be used to drive business actions and responses. Good examples of these sorts of events are a customer placing an order, an equity trade being filled, or a call being received in a call center. These sorts of events are central to the business processes embedded in our applications and BPM solutions.

However, many raw events do not easily correspond, in a one-to-one relationship, to recognizable business events. This is because it is really a group of events that collectively, in a certain context, represents the business event of interest to decision makers and operational personnel. A simple example might be the business event of failing to satisfy a contractual agreement to fulfill orders within one week. Identifying this event would require correlating order creation time with order shipment time and understanding contractual requirements.

More generally, CEP is about describing a pattern of events you want to watch for and then generating a subsequent event, sometimes described as a composite event, when that pattern is satisfied. The pattern not only describes the relationship between events in terms of related identifiers, or keys, but, as a practical matter, also identifies the window of time when the pattern is valid.

Simply put, complex event processing is about correlating events to identify the occurrence of a higher-order event—that is, a pattern match. Ideally, what you want is something that can “listen” to a flow of events as they happen and then react. From this description, you can tell that this is some sort of automated, “background” processing.

Before going on, let’s get a couple of quick definitions out of the way, to try to avoid confusion. What is the difference between an event and a message? Rather than rehash debates more fully presented in other technical venues, this article uses these terms as follows: An event is something significant that is taking place. A message—especially in the context of BPM, enterprise application integration (EAI), or message-oriented middleware (MOM)—is a carrier of event information. Not all messages are necessarily events.

How Is CEP Distinct from BPM?

Most BPM tools can correlate messages and transactions within the processes described by those tools. Why is CEP needed, then? The primary reasons are efficiency, modeling ease and ongoing maintainability. Let’s look at why it makes sense to separate CEP from BPM and what additional capabilities it provides.

BPM tools are focused on doing work. When you join, or correlate, messages in a BPM process, your expectation is that the process will almost always receive all the messages it needs in order to complete the task. Failure to receive all expected messages is likely to be a rare exception case. CEP, on the other hand, deals primarily with the opposite situation, in which you are inspecting numerous messages, only a very few of which, in a particular pattern, are of interest. Understanding this distinction, you can optimize the tools for each use case. Although BPM can deal with exception cases, it is an inefficient use of BPM resources if these exception cases become the norm rather than the exception.

A second difference has to do with CEP simplifies defining and recognizing patterns among many different event situations. Some of these patterns may be fairly simple — for example, we’re interested if event A and event B both happen — however, the real world is rarely so simple. Even in that simple example, it is likely that there exists some sort of “valid window” rather than it being an unbounded situation. Using CEP means dealing easily with such real world situations, because CEP provides additional constructs such as identifying the “valid window” for correlation and invalidation, nonevent events (want to know when event B does not happen after event A), repeated events, and the like. Most correlation capabilities in BPM tools focus on simple joins and hierarchical relationships that make accommodating these constructs difficult. CEP tools are focused on providing modeling and languages to make these constructs easy.

Third, for ongoing maintainability, it makes sense to separate complex-event processing from simple-event processing and the processes associated with those simple events. In this regard, you can think of CEP as turning complex patterns and sets of events into a simple event, which is then easily dealt with by BPM tools or other tools such as BAM. Abstracting the logic associated with complex event patterns into its own, cohesive unit makes it easier to catalog these patterns, add more patterns, or change these patterns without having to modify BPM processes associated with them.

Additionally, many emerging CEP tools also deal with an interesting challenge of event processing: accommodating high-volume, short-duration event flows, versus long-running situations. As you might imagine, these very different use cases can push you to very different solutions, such as in-memory processing versus database-centric solutions. The result is often that it becomes difficult afterward to combine events from the different environments. Many CEP tools can abstract the modeling of the processing from the actual implementation, meaning that designers can focus on describing the pattern they are looking for, rather than worrying about the specific characteristics of the event, or the event flow, itself. The CEP runtime environment doing the actual processing can then be optimized to accommodate different event duration requirements, transactional needs, or persistence/archiving requirements.

Most of what has just been described is about how CEP “preprocesses” events for BPM with the output event being used to initiate a BPM process. However, bear in mind that processes themselves emit events, either as output or as intermediate status events. Therefore, CEP also provides a mechanism for linking processes in situations where a process does not always deterministically follow other processes or for monitoring business processes to identify aggregate, unique, or anomalous situations. The second use case is the most frequent in the context of BAM.

Benefits of CEP for BPM

The combination of CEP (often as part of a larger BAM solution) with BPM yields a complete solution toolkit that can more easily combine events, services, and processes to create very adaptable, responsive, and flexible solutions.

In the telecommunications industry, a heavy user of workflow; MOM/EAI; and, increasingly, BPM solutions, the focus is on two areas: how to instrument for critical scenarios and how to be more flexible and agile as business needs change. This industry needs to easily instrument for specific situations, such as combinations of network and customer support escalations and failure to fulfill provisioning requirements, that are not necessarily accommodated in the core processes but are critical and require rapid response when they occur. These situations are often in flux, because new ones keep cropping up and existing ones need modification. In many cases, the “rich response”—a defined escalation workflow—already exists but the organizations are not detecting the situations until it is too late to head them off or quickly mitigate them. CEP provides a mechanism that lets organizations more quickly, or even proactively, detect and respond to issues, thereby reducing support costs and, in some cases, avoiding contractual penalties while improving overall customer satisfaction.

CEP ensures that procedures and processes are more effectively used in a timely and consistent fashion. In many cases, organizations have the right ones in place but are not fully utilizing them—they do not have the situational awareness to bring them to bear in a timely fashion, because they detect situations that require attention either too slowly or not at all. CEP mitigates both of these issues, by ensuring that recognition of particular situations is consistent and timely.

CEP enables greater leverage of existing processes. Its separation of processes into “when something should be done” and “what work should be done” results in a more adaptable and agile environment. As you realize that there are additional situations that should cause some activity to take place, it is easy to just add them to the list. CEP provides a mapping mechanism between many different situations (a group of events) and a single business event trigger that makes something happen. Similarly, if you realize that a particular business event should cause other activity to occur, you can reuse that event as the initiator for other processes.

Essentially, rather than being limited to having people initiate processes or workflows or just having simple transactions or events trigger processes, CEP gives organizations a wider perspective and enables them to automate the identification of real-world situations. CEP can deal with the noise of the constant stream of low-level events, filtering out just those patterns that have business significance.

Conclusion

In summary, CEP provides a mechanism for easily describing and identifying specific patterns of events. Organizations are already flooded by numerous events that originate from systems, applications, sensors, and people. Existing applications and processes are effective in doing work in response to explicit, straightforward events, but it is often the combination of events in particular patterns that collectively signify business events with business significance. The events themselves already exist, but it is just necessary to collect and filter them to find those of interest. CEP extracts actionable business events out of the background event noise pervading organizations.

The value for BPM solutions is that CEP turns a complex group of events into a simple, composite event that existing BPM tools can easily leverage. This allows CEP to be incrementally added into environments that have, or are planning, BPM capabilities. More important, it abstracts logic about “when something should be done” from the business process logic of “what work should be done.” It provides for an architecture in which events, services, and processes become building blocks that can be assembled, or “wired together,” as necessary. This creates a more responsive and adaptable overall infrastructure and gives organizations more agility to meet changing business needs.