Data integration maturity levels and the future of data integration

Read about current data integration maturity levels, data integration trends, how data integration technologies have evolved and the future of the data integration market.

Lean IntegrationIn this excerpt from Lean Integration: An Integration Factory Approach to Business Agility, readers will learn about data integration maturity levels and how data integration technologies have evolved over the years. 

Readers will also get an overview of how to start implementing a lean integration project.

Table of Contents
Why lean integration is important to data integration systems
Data integration issues that a lean integration strategy can solve
Data integration maturity levels and the future of data integration

Integration Maturity Levels

Another way to look at integration is to examine how integration technologies and management practices have evolved and matured over the past 50 years. Figure 1.2 summarizes four stages of evolution that have contributed to increasingly higher levels of operational efficiency. Hand coding was the only technology available until around 1990 and is still a common practice today, but it is gradually being replaced by modern methods. The movement to standard tools, more commonly known as middleware, began in the 1990s, followed by industry consolidation of tool vendors during the first decade of the 2000s, resulting in more “suites” of tools that provide the foundation for an enterprise integration platform.

As we look to the future, we see the emergence of the Integration Factory as the next wave of integration technology in combination with formal management disciplines. This wave stems from the realization that of the thousands of integration points that are created in an enterprise, the vast majority are incredibly similar to each other in terms of their structure and processing approach. In effect, most integration logic falls into one of a couple of dozen different “patterns” or “templates,” where the exact data being moved and transformed may be different, but the general flow and error-handling approach are the same. An Integration Factory adds a high degree of automation to the process of building and sustaining integration points. We believe the Integration Factory, described in detail in Chapter 3, will be the dominant new “wave” of middleware for the next decade (2010s).

Management practices have also evolved from ad hoc or point-in-time projects, to broad-based programs (projects of projects), to Integration Competency Centers (ICCs), and now to Lean Integration. A major paradigm shift began early in the current century around the view of integration as a sustaining practice. The first wave of sustainable management practices is encapsulated by the ICC. It focused primarily on standardizing projects, tools, processes, and technology across the enterprise and addressing organizational issues related to shared services and staff competencies. The second wave of sustainable practices is the subject of this book: the application of Lean principles and techniques to eliminate waste, optimize the entire value chain, and continuously improve. The management practice that optimizes the benefits of the Integration Factory is Lean Integration. The combination of factory technologies and Lean practices results in significant and sustainable business benefits.

The timeline shown on the bottom of Figure 1.2 represents the period when the technology and management practices achieved broad-based acceptance. We didn’t put a date on the first evolutionary state since it has been with us since the beginning of the computer software era. The earlier stages of evolution don’t die off with the introduction of a new level of maturity. In fact, there are times when hand coding for ad hoc integration needs still makes sense today. That said, each stage of evolution borrows lessons from prior stages to improve its efficacy. We predict that Lean practices, in combination with past experiences in project, program, and ICC practices, will become the dominant leading practice around the globe during the 2010s.

In Part III of this book we refer to these four stages of evolutionary maturity when discussing the eight integration competency areas. The shorthand labels we use are as follows:

  1. Project: disciplines that optimize integration solutions around time and scope boundaries related to a single initiative
  2. Program: disciplines that optimize integration of specific cross-functional business collaborations, usually through a related collection of projects
  3. Sustaining: disciplines that optimize information access and controls at the enterprise level and view integration as an ongoing activity independent of projects
  4. Lean: disciplines that optimize the entire information delivery value chain through continuous improvement driven by all participants in the process

We think of this last level of maturity as self-sustaining once it becomes broadly entrenched in the organization.

We don’t spend much time in this book discussing project or program methods since these are mature practices for which a large body of knowledge is available. Our focus is primarily on sustaining practices and how Lean thinking can be applied to achieve the highest levels of efficiency, performance, and effectiveness.

Economies of Scale (the Integration Market)

As stated earlier, the benefits of Lean are economies of scale and reduction in variation. As a general rule, doubling volume reduces costs by 15 to 25 percent, and doubling variation increases costs by 25 to 35 percent. The ideal low-cost model, therefore, is maximum standardization and maximum volume. But how exactly is this accomplished in a Lean Integration context?

A core concept is to view the collection of information exchanges between business applications in an enterprise as a “market” rather than as a bunch of private point-to-point relationships. The predominant integration approach over the past 20 years has been point-to-point integration. In other words, if two business systems need to exchange information, the owners and subject matter experts (SMEs) of the two systems would get together and agree on what information needed to be exchanged, the definition and meaning of the data, the business rules associated with any transformations or filters, the interface specifications, and the transport protocol. If anything needed to change once it was in operation, they would meet again and repeat the same process.

For a small number of systems and a small number of information exchanges, this process is adequate and manageable. The problem with a hand-coded or manual method is that it doesn’t scale, just as manual methods for other processes don’t scale well. Certainly if a second integration point is added to the same two systems, and the same two SMEs work together and use the same protocols, documentation conventions, and so on, the time and cost to build and sustain the integrations will indeed follow the economy of scale cost curve. But in a large enterprise with hundreds or thousands of applications, if each exchange is viewed as strictly an agreement between the immediate two parties, diseconomies begin to creep into the equation from several perspectives.

Imagine trying to follow a flow of financial information from a retail point-of-sale application, to the sales management system (which reconciles sales transactions with refunds, returns, exchanges, and other adjustments), to the inventory management system, to the general ledger system, to the financial reporting system. In a simple scenario, this involves four information exchanges among five systems. If each system uses different development languages, protocols, documentation conventions, interface specifications, and monitoring tools and was developed by different individuals, not only will we not receive the benefits from quadrupling volume from one integration to four, but we will in fact increase costs.

This example reflects the two largest factors that drive diseconomies: the cost of communication between teams and duplication of effort. Additional factors can drive further diseconomies, such as the following:

  • Top-heavy organizations: As organizations get larger and add more layers of management, more and more effort needs to be expended on integrated solutions that require collaboration and agreement across teams that each play a narrow functional role.
  • Office politics: Disagreements across teams are a result of different motivations and agendas, usually a result of conflicting goals and metrics but also sometimes caused by the “not invented here” syndrome.
  • Isolation of decision makers from the results of their decisions: Senior managers may need to make decisions, such as how much of a budget to allocate to a given group, without a clear picture of what the group does and what value it brings to the organization.
  • Slow response time: Delays are caused by multiple handoffs between teams or by queuing requests for information or support from other groups.
  • Inertia: People are unwilling to change or are opposed to standards.
  • Cannibalization: Limited resources (such as SMEs in specific business domains) are consumed for project B, slowing down progress on project A.

The degree of integration variation in many organizations is staggering in terms of both the variety of technology that is used and the variety of standards that are applied to their implementation. That is why most organizations have a hairball – hundreds or thousands of integrations that are “works of art."

The alternative is to view the need for information exchanges across applications as a market economy, and to serve the market with an efficient shared-services delivery model in order to gain economies of scale. For example, multiple groups within an organization may perform similar activities but do so with varying degrees of efficiency and consistency. Centralizing the activities makes it much easier to standardize and streamline the work, thereby reducing the cost per unit of work while improving the quality and consistency.

The two graphs in Figure 1.3 are borrowed from the field of economics and show the relationships between costs and volumes. These graphs reflect the well-understood laws of diminishing returns and economies of scale. The chart on the left reflects a manual or low-tech operation, such as when information exchanges are developed using custom hand-coded integration solutions. In this scenario, there are few (if any) capital costs since existing enterprise tools such as COBOL, Java, or SQL are used to build the integration. The overall average cost per integration initially falls as the individuals doing the work gain experience and are able to share some of their experience and knowledge, but then at some point it starts to increase as diseconomies emerge. In terms of the marginal costs (i.e., the incremental cost for each additional integration), initially the curve is somewhat flat since the first integration developer can develop a second or third integration with a similar effort. The average cost also falls initially since the fixed costs of the developer (hiring costs, office space, desktop PC, etc.) are amortized over more than one integration. As the volume of integrations increases, however, the marginal costs increase on an exponential basis, and the average costs begin to increase as more and more diseconomies emerge from the increasing complexity and number of unique integration points.

The chart on the right of the figure shows the cost curve as a result of a capital investment in tools and infrastructure (such as an Integration Factory) to largely automate and standardize the development effort. Note that in this scenario, the marginal costs are small and constant. For example, it might cost Microsoft $5 billion to develop a new version of Windows, but once developed, it costs just pennies to make another electronic copy. The marginal cost per copy of Windows is essentially the same whether 1 copy or 1,000 copies are made, but the average cost drops significantly and continuously in this scenario as volume increases and as the up-front fixed costs are amortized over more and more units.

The key challenge for organizations is to determine at what level of integration complexity do diminishing returns begin to emerge from manual hand-coded solutions, and how much capital investment is warranted to achieve the time and cost advantages of a high-volume Integration Factory. The answer to this will become clearer in Parts II and III, where we discuss the Lean principles related to continuous improvement, mass customization, and process automation, and the financial management competency area.

Getting Started: Incremental Integration without “Boiling the Ocean”

Parts II and III of the book provide detailed and specific advice on how to implement a sustainable Lean Integration practice, but before you dig into the details, it is important to understand the approach options and related prerequisites.

There are two fundamental implementation styles for Lean Integration: top-down and bottom-up. The top-down style starts with an explicit strategy with clearly defined (and measurable) outcomes and is led by top-level executives. The bottom-up style, which is sometimes referred to as a “grassroots” movement, is primarily driven by front-line staff or managers with leadership qualities. The top-down approach generally delivers results more quickly but may be more disruptive. You can think of these styles as revolutionary versus evolutionary. Both are viable.

While Lean Integration is relevant to all large organizations that use information to run their business, there are several prerequisites for a successful Lean journey. The following five questions provide a checklist to see if Lean Integration is appropriate to your organization and which style may be most suitable:

1. Do you have senior executive support for improving how integration problems are solved for the business?
Support from a senior executive in the organization is necessary for getting change started and critical for sustaining continuous improvement initiatives. Ideally the support should come from more than one executive, at a senior level such as the CXO, and it should be “active” support. You want the senior executives to be leading the effort by example, pulling the desired behaviors and patterns of thought from the rest of the organization. It might be sufficient if the support is from only one executive, and if that person is one level down from C-level, but it gets harder and harder to drive the investments and necessary changes as you water down the top-level support. The level of executive support should be as big as the opportunity. Even with a bottom-up implementation style, you need some level of executive support or awareness. At some point, if you don’t have the support, you are simply not ready to formally tackle a Lean Integration strategy. Instead, just keep doing your assigned job and continue lobbying for top-level support.

2. Do you have a committed practice leader?
The second prerequisite is a committed practice leader. By “committed” we don’t mean that the leader needs to be an expert in all the principles and competencies on day one, but the person does need to have the capability to become an expert and should be determined to do so through sustained personal effort. Furthermore, it is ideal if this individual is somewhat entrepreneurial, has a thick skin, is customer-oriented, and has the characteristics of a change agent (see Chapter 6 on team empowerment for more details).

If you don’t have top leadership support or a committed practice leader, there is little chance of success. This is not to suggest that a grassroots movement isn’t a viable way to get the ball rolling, but at some point the bottom-up movement needs to build support from the top in order to institutionalize the changes that will be necessary to sustain the shift from siloed operations to integrated value chains.

3. Is your “Lean director” an effective change agent?
Having a Lean director who is an effective change agent is slightly different from having one who is “committed.” The Lean champion for an organization may indeed have all the right motivations and intentions but simply have the wrong talents. For example, an integrator needs to be able to check his or her ego at the door when going into a meeting to facilitate a resolution between individuals, who have their own egos. Furthermore, a Lean perspective requires one to think outside the box – in fact, to not even see a box and to think of his or her responsibilities in the broadest possible terms. Refer to the section on Change Agent Leadership in Chapter 6 for a description of essential leadership capabilities.

4. Is your corporate culture receptive to cross-organizational collaboration and cooperation?
Many (maybe even most) organizations have entrenched views of independent functional groups, which is not a showstopper for a Lean program. But if the culture is one where independence is seen as the source of the organization’s success and creativity, and variation is a core element of its strategy, a Lean approach will likely be a futile effort since Lean requires cooperation and collaboration across functional lines. A corporate culture of autonomous functional groups with a strong emphasis on innovation and variation typically has problems implementing Lean thinking.

5. Can your organization take a longer-term view of the business?
A Lean strategy is a long-term strategy. This is not to say that a Lean program can’t deliver benefits quickly in the short term – it certainly can. But Lean is ultimately about long-term sustainable practices. Some decisions and investments that will be required need to be made with a longterm payback in mind. If the organization is strictly focused on surviving quarter by quarter and does little or no planning beyond the current fiscal year, a Lean program won’t achieve its potential.

If you are in an organizational environment where you answered no to one or more of these questions, and you feel compelled to implement a Lean program, you could try to start a grassroots movement and continue lobbying senior leadership until you find a strong champion. Or you could move to another organization. There are indeed some organizational contexts in which starting a Lean program is the equivalent of banging your head against the wall. We hope this checklist will help you to avoid unnecessary headaches.

Lean requires a holistic implementation strategy or vision, but it can be implemented in incremental steps. In fact, it is virtually impossible to implement it all at once, unless for some reason the CEO decides to assign an entire team with a big budget to fast-track the implementation. The idea is to make Lean Integration a long-term sustainable process. When we say “long-term” we are talking about 10 to 20 years, not just the next few years. When you take a long-term view, your approach changes. It certainly is necessary to have a long-term vision and plan, but it is absolutely acceptable, and in many respects necessary, to implement it incrementally in order to enable organizational learning. In the same way, an ICC can start with a small team and a narrow scope and grow it over time to a broad-based Lean Integration practice through excellent execution and positive business results.

Table of Contents
Why lean integration is important to data integration systems
Data integration issues that a lean integration strategy can solve
Data integration maturity levels and the future of data integration

More about this book and others like it...


Dig Deeper on Data integration

Business Analytics
Content Management