Articles

Moving Beyond Composite Applications to the Next Generation of Application Development: Automating Exception-Rich Business Processes

The market for enterprise applications is maturing. Today, enterprise software packages are available to automate almost every aspect of a business. Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Supply Chain Management (SCM), Materials Resource Planning (MRP), and operational management suites are available from multiple vendors.

As enterprise applications rapidly become commodities, a company’s ability to differentiate itself resides in the exception-rich business processes that exist between applications rather than inside the applications themselves. Enterprise software development has moved away from the development of individual applications, re-focusing on the effective integration of these applications into a cohesive whole.
 
Composite applications are at the heart of current approaches to enterprise integration and vendors are integrating the approach into the heart of their product strategies, as discussed by Dan Woods in his book Packaged Composite Applications (O’Reilly & Associates, ISBN 059600520, June 2003). A composite application addresses a gap between existing enterprise applications, integrating functionality across a range of applications to create a task-specific solution—letting the solution specify the steps of a business process rather than requiring the users to mediate between each application. Existing applications from a company’s portfolio are cracked open, exposing as Web Services the suite of basic services—the business transactions—that each legacy application supports. Any additional functionality required is developed as new services within an overall Service-Oriented Architecture (SOA). Finally, the set of services are strung together using workflow tools and techniques (such as Business Process Execution Language [BPEL]) to create higher-level business process services to support the end-to-end business processes required. These services script together the functionality of business transactions to deliver solutions targeted to a specific user, business need, or process (see Figure 1).
 
Composite applications have already harvested much of the low-hanging fruit by providing companies with a suite of targeted solutions that consolidate existing enterprise application functionality, helping the business streamline operations and focus on the problem at hand: delivering value to customers and shareholders. Rather than interacting with a range of systems to carry out their job, users are now provided with a focused user interface that reduces the effort required to carry out their daily work, reducing cost, and improving overall quality of service. However, existing service-based approaches are reaching their limits and new tools and techniques are needed to drive this growth. New development efforts must move up the business value chain and deliver solutions to the exception-rich business problems previously beyond IT’s reach.
 
From Consolidation to Automation
 
With the first generation of composite applications rapidly maturing, it’s time for the next step in the evolution of the technology. This step will build on the consolidated business processes provided by composite applications, automating key decision and exception management processes to deliver applications which will—for the first time— truly capture the dynamic nature of a business process as a collection of cooperating, deliberating, and negotiating services.
 
Automation lets us bring the benefits of software to these previously manual processes. A software-based solution can look across the enterprise to create globally optimal solutions where manual processes are restricted to locally acceptable solutions, constrained by the limited data and time available to human operators. To do this, we need to add a new layer to the technology stack (see Figure 2)—agents and business Webs—that contains technologies and approaches capable of automating the exception-rich business processes and deliberations currently the domain of human operators. Previously, we had a suite of business transactions (Web Services) coordinated by clerks (BPEL); now we introduce a layer of agents capable of managing business exceptions and planning the operations in the layer below. This provides us with a simple three-layer technology model that lets us associate specific technologies and approaches with different types of functionality in the final solution.
 
A simple analogy is to compare the different layers to the roles in a simple shipping business process. If you were to walk into a shipping office to ship a package, then the three layers would be represented by the following business entities:
 
• Shipping business transactions represent the basic business transactions required to fulfill the business process. These include common business operations such as creating a waybill, scheduling a pick-up, etc.
• You interact with a shipping clerk, who is responsible for shipping business processes—such as filling out and validating the waybill, calculating the charge or booking a pick-up—created by stringing together the shipping business transactions. However, the shipping clerk can’t cope with business exceptions or complex business decisions.
• A shipping agent is responsible for planning a shipment, and dealing with any business exceptions that arise. Whereas the shipping clerk knows how to carry out a linear business process, the shipping agent understands the process’s purpose and how the process is integrated into the business, and can construct new business process from existing business transactions supported by the shipping clerk.
 
These new agent services require tools and techniques capable of supporting the development and maintenance of complex, exception-rich business processes. One obvious candidate is agent technology, as outlined by Nicholas Jennings in his 2001 article, “An Agent-Based Approach to Building Complex Systems,” which ran in Communications of the ACM (Volume 44, Number 4). Agent technology is a large field that covers a wide range of disciplines; though, at its core, the technology is simply inspired by the flexibility and sophistication seen in living systems. The technology draws inspiration from a wide variety of fields, including philosophy, physiology, biology, game theory and mathematics, logic, linguistics, economics and computer science. To learn more, see the book by Michael Wooldridge, Introduction to Multi-Agent Systems (John Wiley & Sons, ISBN 047149691X, June 2002).
 
Agent technology’s anthropomorphic nature makes it a good fit to the development of agent-level services. The exception rich, goal-based, trial-and-error nature of real-world business decisions can be directly supported by modeling these decisions with goal-directed technology. This view was highlighted in an article titled “Procedural Knowledge” in a special knowledge representation issue of Proceedings of the IEEE (74:1383–1398, M. Georgeff and A. Lansky, 1986).
 
In the agent layer, business exceptions are no longer exceptional; they’re simply alternative ways of achieving a goal. Business processes consist of a series of decisions made by different business roles. The roles are formed into networks of negotiating, cooperating, agents working toward a common goal. Multi-agent systems let us use tools and techniques from agent technology to directly capture and implement the roles, their relationships, and the communication between them.
 
A Look at Logistics
 
Logistics management is typical of the style of business problem for which Web Service-enabled composite applications are suited. The current trend in logistics is to divide support between planning applications, which compute production plans overnight, and execution applications, which manage the flow of events in an operational environment (see Figure 3). This disconnection forces users to deal with business exceptions (lost shipments, for example), manually resolving the problems by directly updating the execution and planning applications.
 
We can begin to address this problem by leveraging Web technology to create a heterogeneous composite application involving all participants in the process, providing a complete Third-Party Logistics (3PL) solution. The new solution fits between the existing applications, integrating them to create a new end-to-end process that extends the applications rather than replacing them outright. The final composite solution connects the planning and execution applications, giving users a single unified view into the logistics pipeline (see Figure 4). This consolidated logistics solution greatly simplifies the task of identifying and correcting business exceptions (e.g., missing shipments or stock shortages) as they occur.
 
From Automation to Optimization
 
Today’s logistics applications take a simple black-box view of optimization: Production is matched to expected demand and then fed into a set of pre-defined delivery pipelines, with business exceptions and deviations from the plan requiring human intervention. Once a shipment has entered a pipeline, we can only delay its progress, and it’s impossible for users to optimize across pipelines by diverting underused capacity in one to cover a deficiency in another. The result is a logistics network that contains excess stock, as the disconnection between the planning and execution applications restricts the plan to managing production on a daily basis: tuning the input to the pipeline but not movements through it.
 
One approach to improving the efficiency of the logistics network is to introduce a set of services supporting the real-time planning and exception management functions required to manage operation of the logistics pipeline (see Figure 5). These exception-rich roles were previously the domain of users, but leveraging agent technology lets us integrate them into the company’s SOA, creating an enhanced solution capable of automating the dynamic behavior required to support real-time logistics management.
 
Agent technology is a good fit for these services as each service implements processes and heuristics representing the goal-based, trial-and-error approach users used to manually solve these problems in the past; approaches that aren’t cost-effective with conventional technologies and tools. The final solution contains services in all three layers (see Figure 6).
 
The use of agent technology enables a degree of flexibility not possible with conventional approaches. The agent-based services isolate key business decisions behind cleanly defined interfaces, rather than forcing these decisions to be implemented with design patterns that bake business decisions into the solution’s structure, where changing the business decision involves re-factoring the solution.
 
In contrast, agent-based tools and techniques provide a simple, high-level, approach to capturing this goal-based behavior. Adding a new optimization strategy is simply a matter of defining the new optimization process and then adding it to the set of approaches for the agent to consider. The agent manages— at run-time—the trial-and-error process of trying different optimization approaches until a successful one is found.
 
Agent technology has proved to be a powerful tool, enabling the development of more flexible, adaptive and sophisticated solutions than possible with more conventional approaches but its adoption has so far been limited to niche applications. Integrating the technology with a modern service-based architecture lets us deliver enterprise-class applications as networks of dynamic business services, supporting the sophisticated negotiation and exception management necessary to enable this next wave of enterprise software development.
 
Dynamic Business Webs
 
This approach to role automation can be extended beyond the original system boundary to integrate multiple enterprises into a dynamic business Web: a collection of applications and services, representing different companies (or departments), that collaborate in a single market to manage resources and negotiate. Companies can use business Webs to coordinate operations between their various departments, or to automate planning and management with their partners.
 
One example is integrating multiple 3PL solutions into a logistics market (see Figure 7), extending the 3PL logistics management solution beyond the original model to support Fourth-Party Logistics (4PL) operation. A 4PL solution lets us combine multiple 3PL providers into a single heterogeneous virtual logistics network—a dynamic business Web where customers, suppliers and partners can come together to create tailored solutions to business problems in real-time. Each shipment is assigned a route dynamically assembled from one or more individual logistics providers.
 
The composite 4PL application consists of a distributed network of heterogeneous (agent) services using auction-based algorithms to negotiate logistics and shipment routes, dynamically creating virtual supply chains. Previously, these business functions were manual and offline, where today agent technology lets us use algorithms to automate them. The algorithms include advanced features such as multi-item markets, multi-unit markets, trust, coalition formation, contract formation, and breach management. The enhanced application uses this functionality to support, online and in real-time, the dynamic formation and management of virtual supply networks distributed across multiple logistics providers.
 
Each participant in the business Web is implemented with Web Service tools and techniques extended with agent technology while still externally presenting conventional Web Service interfaces. The collection of (agent) services operates as a multi-agent system—each agent acting independently but coordinating its actions with other agents via interaction protocols and agent-based communication design patterns, such as those defined by the Foundation for Intelligent Physical Agents (FIPA) (http://www.fipa.org).
 
Conclusion
 
The rapid commoditization of the enterprise application market has shifted the focus away from the development of these applications, re-focusing our efforts on how to most effectively integrate applications. Current integration efforts are focused on creating composite applications to consolidate application functionality—presenting users with simple task-specific interfaces rather than requiring them to mediate between multiple enterprise applications. As this current integration effort matures, opportunities to consolidate existing functionality will be exhausted and new development efforts will move up the technology chain to provide more advanced behavior. This move will be characterized by a shift from developing composite applications designed to consolidate processes and data to solutions that automate and optimize the process brought into focus by the original composite application.
 
These solutions can form the basis for the creation of dynamic business Webs—virtual solutions that span enterprise applications and even organizations—and dynamically provision the resources they require. Current tools and techniques can’t support this new generation of solutions due to the non-linear, exception-rich nature of the processes they must implement. Agent technology promises to fill this gap by providing the tools and techniques required to extend existing Web Service and SOA-based approaches. The technology integrates cleanly with a modern service-based approach to application development, with the SOA environment providing a sound foundation for the advanced functionality provided by an agent-based solution.
 
Marrying agent technology with Web Service technology results in applications that exhibit a flexibility and robustness not seen with current technologies—creating dynamic business Webs that dynamically adapt to the rapidly changing business environment. Combining the two technologies can provide us with the platform required to effectively integrate commodity enterprise applications into a cohesive whole.

 

About The Authors: Peter Evans-Greenwood and Mark Stason

Writers