Introduction

SOA is an architectural style that can enable the agile enterprise. Unlike conventional IT architectures the style must be transformed into a coherent, actionable structure that goes beyond purely technology concerns and guides day to day activities to ensure that business and technology architectures combine to deliver business agility.

While service concepts are being enthusiastically embraced there is a fundamental problem in that most enterprises are approaching this challenge without adequate structure, policies and repeatable processes. There is something near service anarchy in many corporations as inconsistent, incomplete and duplicative services are delivered by unchanged solution and integration processes. Already some organizations are referring to “legacy services”.

The reason for this widespread sub-optimization is that SOA is not a product or platform that can simply be acquired either off the shelf or through a systems integration project. Each enterprise has to figure out its own unique strategy to respond to specific business challenges. Further each organization needs to implement policies and processes that exert governance to ensure the enterprise capability stays agile.

This report, intended for a general IT and business readership, advises on the urgent need to adopt service specific practices that enable repeatable, measurable and manageable approaches to business process and information systems support. We shall refer to this as Service Architecture and Engineering.

If there is no formal agreement and management of architectural style and principles then SOA is almost certainly a waste of money.

Consistently high levels of business agility will not happen by accident, it will only happen by design and this requires a step change in the execution of architecture and engineering.

The Problem

Various surveys¹ report a majority of enterprises are now adopting SOA in some shape or form. However CBDI Forum’s experience, supported by the surveys, is that the path to SOA is littered with roadblocks and diversions. These are generally not technology issues, more often they relate to the difficulties of selling the benefits of SOA to the CIO and the business; or the impracticality of synchronizing the objectives of enterprise architecture with current business facing projects; or the impossibility of aligning longer term SOA objectives with the business directives to deliver 12 month ROI. We constantly hear the comment that “the technology is easy, it’s the process and organizational issues that are really difficult”. Consequently many enterprises are approaching SOA from a distinctly tactical perspective. Worse, they are justifying the investment and funding in SOA on the promise of agility and reuse, but subsequently doing little to ensure their delivery.

But the more strategic use of SOA principles can evolve rapidly. As enterprises gain experience from early projects they will have the confidence to address wider, more challenging goals and there are organizations demonstrating real progress with advanced service architectures. Last year CBDI Forum reported on BT² that is delivering 18 generalized services that enable significant rationalization and business consistency. More recently CBDI Forum noted Schwab³ has executed on a domain based SOA strategy that moves service responsibility closer to the customer – a good indication of a more mature approach to service architecture.

These examples suggest some organizations have successfully challenged and reengineered their existing business and organizational models.

The root problem for many, if not most organizations is that they are treating SOA as just another technology to be accommodated into their existing practices. They are deploying and using services but there is no structure to their service activity and there is little or no consensus or consistency across the organization on what SOA is, let alone explicit policies and repeatable processes that permit governance. Many organizations are assuming that SOA will simply be grafted onto existing project plans, funding streams, conventional ownership practices, existing responsibilities, processes and policies.

SOA requires a differentiated approach that is more analogous to a manufacturing and assembly environment, where different classes of service have different, repeatable processes that reflect varying requirements for funding, provisioning, assembly and management. We have to move beyond the one size fits all project profile and adopt more mature architecture and engineering processes and practices for delivering and managing IT assets that more closely parallel other engineering disciplines. This is an essential next step to manage outcomes that are not only fit for purpose, but also deliver future flexibility, utility and cost.

Focus on Principles

Curiously, even after some four years of media hype and enterprise deployment, SOA remains an elusive concept. After I spoke at a recent conference one of the other speakers who was presenting his high profile SOA project, came up to me and admitted that his “SOA project” fell very far short of the reference architecture that I had been outlining. There was no policy, and consequently no governance; there was little or no abstraction or architectural consideration of future flexibility. Their SOA aims were really about use of services to undertake integration related activity in a more flexible manner. All really good things to do, but very clearly not embracing the broader aims of SOA.

One reason that SOA remains an elusive concept is that the architectural principles have much broader applicability than the conventional IT architectures that we are more familiar with. For example CICS, Client/Server, J2EE, .NET have quite constrained purpose, and they can be acquired as platforms and tools. But enterprises acquiring an Enterprise Service Bus (ESB)⁴ discover very rapidly that what they have bought is a technology backplane with no instructions on how to deliver the business benefits.

SOA principles are not inherent to any technology, product, platform or tool. They require enterprise specific architectural and engineering thinking to utilize them. Let’s consider an architectural discipline for cathedrals and churches and specifically the well known Gothic style. In its time Gothic architecture introduced new technologies that allowed huge, high cathedrals to be constructed with an elegance that had never previously been possible. The style emphasizes verticality and features almost skeletal stone structures with great expanses of glass, pointed arches using diagonal, ribbed vaults, clustered columns, sharply pointed spires and the amazing flying buttresses. The Gothic style became a widely copied pattern and has been replicated worldwide. Of course each cathedral is a unique instance that uses the Gothic architectural principles, and remains instantly recognizable.

The application of an architectural style is a therefore process of adaptation to suit a specific situation using appropriate engineering techniques to deliver a unique result, that is still recognizable as being based on the key architectural principles. One imagines that builders of Gothic cathedrals down through the ages must have studied the principles with enormous care.

The Agile Character of SOA

Following the deluge of vendor SOA marketing over the past couple of years everyone will be thoroughly familiar with the idea that SOA can deliver agility. But what is agility? Can we measure it? Can we manage it? And will we recognize it when we see it? Can we determine the cost benefit?
It is probably reasonable to say that elegance on a grand scale is to Gothic cathedrals as agility is to SOA. Service Oriented Architecture is an agile style.

This should cause us to stop and think hard about just what is agility, and what might it mean for a particular enterprise or ecosystem. If we can better define the agility needs for a specific enterprise we perhaps have a reasonable chance of knowing what principles are relevant and how we might apply them. The need for business agility for example, is rarely served just by the application of loosely coupled technology.

Just because services are loosely coupled, modular and so on, doesn’t necessarily mean that everything can be in a constant state of change. While this argument is advanced, by people that should know better, the reality for many large enterprises is that change must be managed. Services that are providing core capabilities need to be engineered to be highly stable, whereas other services may be specifically engineered for constant change perhaps because they are identified as supporting an innovative area of the business and need to be highly responsive.

Conversely there will be ecosystems of services that are not managed at all. Discovery and orchestration of services that were designed in isolation provides a profoundly different challenge that is best suited to using ontological mapping and transformation, and a completely different form of management. Yet the principles of SOA should apply equally to both domains.

In practical terms not all principles will apply equally to all services. Clearly not all services will be standard. There will be widely varying levels of abstraction. However it might be expected than an enterprise embracing SOA would be looking to utilize all the principles where they are applicable.

SOA Principles

Once the enterprise, or ecosystem, has understood the relevance of the style it is then in a position to clearly identify how the relevant principles should be applied. If we are even slightly confused about the relevance of the style then we are going to be in trouble with interpreting principles.

Looking around one can observe wide inconsistency in guidance on principles. This is not the place to embarrass anyone here, but a quick Web search on SOA principles will demonstrate the problem. This inconsistency seems to arise because the various analysts, authors and advisors are not adopting an architectural perspective along the lines discussed. Rather they are embracing a wide range of fragments of guidance ranging from “SOA is REAL” to “a successful SOA is always in flux”, neither of which are terribly helpful.

This disparity is at the heart of the confusion around “what is SOA”. If there is widespread lack of consistency and understanding of what the core principles are, it shouldn’t be a surprise that while 90% of projects are claiming to be doing SOA in many cases a cursory examination reveals they are complying with a minority of the essential core principles.

Figure 1 provides a simple view of the SOA principles. SOA is about separation and the principles provide a basis for measuring the intrinsic separation. The principles also provide the basis for developing process and governance requirements.

Figure 1 – SOA Core Principles

Most SOA projects conform with a minority of the core principles. Most are attempting to address some level of loose coupling because this is the obvious place that agility may be achieved. However the extent to which the loose coupling is built into the business design is likely to be more variable. An engineered approach to service architecture would review the broader range of principles and patterns to develop a practical design that delivers a level of agility that is fit for purpose and makes economic sense. Considerations may include:

• Standardization is something that is just not talked about by many people. Forget reuse, this is a programmers view of the world. Standardization is a business matter and relates to how an enterprise manages its business processes, its data and application portfolio. Standardization is required because the business demands consistency, not because IT desires reuse.
• Abstraction is the most powerful of tools in the agility toolbox. A small amount of work to generalize a service specification can allow the service to support many different contexts, for example product, channel or geographic data and rules may be abstracted to allow a common capability to be used in a consistent manner across an enterprise or ecosystem, or to allow support for future business change within minimal or no effort.
• Composability is again a hugely powerful technique that takes advantage of the fractal nature of SOA that allows hierarchies or assemblies to be constructed based on more common, standardized services at the lower layers that are increasingly specialized at the higher layers.
• Modularity is a concept that can be implemented at many levels in an SOA. Relative dependency and modularity should be determined in the business model and applied to the business processes, services and components. In the early stages an architect should be looking to reduce dependency so that the horizon of change can be predicable, measured and minimized. But as the portfolio is more widely based on service interfaces that make the underlying applications more transparent there will be many opportunities to componentize at all levels of the architecture with considerable benefits of increased agility and reduction in cost.
• Virtualization is an important part of the SOA. The basic service concept, with or without web services, provides a high level of transparency of the underlying resources, providing the loose coupling has been properly implemented and there are no design or platform dependencies established by the service consumer or provider. The virtualization then provides opportunities for the provider and consumer to act independently and to have different life and upgrade cycles with consequent increased agility and response to change.

Life Cycle Process Discontinuity

A delivery and management life cycle process relevant to services must be based on a deep appreciation of the core principles. Communication of how the principles support business objectives and goals is critical to improved business agility because changes in business model or process will only happen when there is clear justification and business case.

In the early stages of SOA there is going to be considerable resistance to change that is based on relatively untried thinking, especially when it is counter to current best practice. For example:

• Project managers work aggressively to reduce external dependencies. They are uncomfortable with increasing third party suppliers.
• There is generally low levels of trust across an organization., Shared endeavors have a track record of failure.
• Reuse of common services or components often leads to duplication of the core capability because it is too difficult to manage common requirements and multiple concurrent release support.
• Agile projects develop localized architecture on an iterative basis.

These challenges are extremely common and addressed in isolation will be difficult to overcome. They can only be resolved by understanding how the solution supports important core business strategies such as business consistency, time to market etc.

When this is properly communicated the necessary organizational and process changes will be more easily accomplished. Table 1 provides a mapping of how each of the core principles may enable improved business outcomes and the business model and process discontinuities that will need to be addressed. This analysis suggests that the organization can take a systematic and engineered approach to understanding the process and organizational change based on transition to new principles. Of course this needs to be thoroughly worked through in context with each organization.

Table 1 - Principle Outcomes Need Discontinuity

The New Architecture & Engineering Meta Model

A further challenge or opportunity for misunderstanding is that SOA doesn’t change everything that we do today. Far from it! SOA is an overlay on existing architectural practices and IT delivery processes. Of course the evolutionary nature of SOA is a clear strength, we can apply the architectural principles to existing IT assets, but it does mean that we have to be especially careful to communicate the delta and to make sure that the status quo doesn’t persist because of existing momentum.

A sensible place to start is to understand the meta model underlying SOA. With a rigorous metadata driven approach it is easier to understand the deliverables delta and the processes that need to be put in place to support them.

In this short report it is not appropriate to include a detailed treatment of the SOA meta model, but Figure 2 provides a highly simplified view of part of the model. The meta model is a detailed definition of the artifacts that are needed to manage the entire life cycle of a service in such a manner that we can establish appropriate standards, techniques, tools and policies that ensure there is complete traceability from the business perspective through to the executable service.

Figure 2 – Highly Simplified SOA Meta Model (Partial)

The New Reference Architecture and Life Cycle

For the first time in the history of IT we have with service a first order construct (formal artifact) that is consistent from the business perspective through to the deployed capability. This common service perspective shown in Figure 3 allows us to have direct connect between the business process and the automation unit(s) and the deployed software. It allows us to have genuine business activity monitoring and to dynamically alter resources and business policies used by the invokable business service. If the service has been specified and provisioned in conformance with the architectural principles there is the opportunity to exert governance over the execution of the service that is directly linked to the original business requirements and policies.

The requirement for formal governance over the design and implementation of services demands that there is some new level of formality in the way architecture activity is undertaken and services provisioned and used. No one wants to implement bureaucracy for its own sake, but if the service architecture principles are to be adhered to there needs to be some appropriate way to define the architectural policies and to allow them to be both communicated and reviewed for compliance.

Figure 3 – The SOA Abstract Model

Figure 4 illustrates how the SOA meta model views and artifacts can be mapped to a deliverable structure that spans the abstract service, implementation and deployment views. Based on the Zachman Framework⁵ the framework has been developed by CBDI Forum⁶ to provide a practical set of reference models. Each of the cells of the reference model are placeholders for policies, standards, templates and compliance review requirements and provides a specification for provisioning of automation platforms and tools.

The reference architecture also provides a structured basis for engineering an appropriate life cycle process in order to manage the governance reviews. Figure 5 illustrates a simplified view of the service life cycle model⁷ that defines various stages in the life of a service, and identifies the activities that move the service between these stages. This model represents a crucial input to the design of SOA development processes and the secure management of software services.

Figure 4 – SOA Reference Architectures

Figure 5 – The Service Life Cycle

The New IT Process

The IT process needs to evolve to establish a more rigorous architectural and planning level framework that both coordinates the development of common, shared services and provides just enough guidance to allow solution delivery projects to use the shared services while retaining the appropriate degree of freedom to innovate. The SOA Reference Architectures providing the policy backplane guides the relationship between the many moving parts that allows change to take place more easily, with a lower impact across the enterprise.

The SOA based process needs to be inherently distributed, and in no way centralized or top down.
Figure 6 illustrates the major building blocks of the service architecture and engineering process. There is a clear separation of shared infrastructure and business service supply that meets the demand created by the agile business facing solution delivery environment.

Figure 6 – Service Architecture & Engineering Building Blocks

The new IT process components include:

• The Adoption Roadmap framework defining the capability/maturity required to successfully deliver and manage all aspects of the SOA based environment across a highly distributed enterprise, providing a basis for planning, managing, measuring and monitoring the organizational and technology infrastructure capability.
• Reference Architectures defining policies, standards, patterns, tools and governance requirements based on an integrated meta model spanning the service, implementation and deployment views.
• A Service Portfolio Planning framework providing a defined approach to identifying and clustering services and capabilities that conform to agreed policy. The Service Portfolio Plan (SPP) is typically developed progressively in line with enterprise and Line of Business (LOB) strategies and is a key mechanism enabling optimization of the SOA into LOB projects and programs.
• A Service Provisioning process defining a rigorous approach to service specification that drives a high quality implementation, procurement, testing and certification process.
• A service customization and assembly framework providing patterns and techniques for customizing and specializing common shared services to meet diverse needs of disparate business units while maximizing appropriate levels of business data and rules consistency.
• Delivery and management of the Operational Infrastructure supporting and managing service execution including security patterns.
• The Service Life Cycle Infrastructure implementing the entire service life cycle managing the IT processes transitioning services through state changes from Planned to Retired including defined governance requirements by state change.
• The service demand side processes including business modeling, business process improvement and solution delivery.

It should be noted that transition to SOA is a medium term program. CBDI Forum advise that the evolution of project patterns and ambitions should be paced with appropriate process support. The adoption roadmap process is key to ensuring a balance is maintained between capability and project demands.

The New IT Business model

The rigorous service specification allows real separation and independence of service provider and consumer and this simple model is going to have profound impacts on IT solution delivery.

For example:

• Enterprise buyers will increasingly select enterprise application based services as part of their own architecture. Where the third party services do not fit the customer business process they will implement the services underneath customizing facades, or replace with alternatively sourced services.
• Service provisioning will increasingly be managed as a specialized activity using individuals who are skilled in creating highly effective abstract services that can support many process contexts.
• As the critical mass of services grows solution and process delivery will increasingly become an assembly process, where standardized services are specialized into specific line of business requirements while maintaining the common functionality to deliver required enterprise consistency. The assembly process will require a quite different set of tools and skills to the provisioning task.
• Business process design will increasingly include demand for service support. The business process specification will include policy statements that drive IT governance.
• Demand for abstract services will need to be articulated as longer term demand. This will require the business to define requirements for longer term business process change.

The New Economic model

Economics is an important aspect of every well engineered process. Many of the less successful SOA projects today are the result of continuing to utilize unchanged and inappropriate economic models.

At the heart of the SOA approach is a profound change in the way IT solutions are delivered which represents a new economic model. As with any economic model our intent is to illustrate principles in a simplistic manner that can be applied to real world situations.

Today the predominant solution delivery model in major enterprises is “Build or Buy to Order” based on Line of Business demand, driven by the practicalities of demand and supply, ownership and funding models. Figure 7 shows a highly simplified picture of this model where LOB projects inevitably create an increasing integration and maintenance cost. We note that in practice, in many organizations if not the majority, the integration costs are subsumed into the LOB project costs, because the integration effort is an integral part of project led demand. This tends to under-estimation of integration costs.

Figure 7 – The SOA Economic Model

In contrast with the Build or Buy to Order, the SOA environment is inherently a “Manufacturing and Assembly Model” in which standard services are delivered as part of a shared service investment program that are then used in an assembly based response to specific demand. The manufacturing layer provisions abstracted, generalized services that can support many situations and contexts. The standard services are the equivalent of the automotive industry subframe, drive train and other core components.

The assembly layer takes advantage of the fractal nature of SOA and (dependent upon policy) specializes and extends the standard services in order to meet specific requirements of individual business processes, geographies and product groups. The enterprise becomes inherently agile because the assembly process can respond extremely rapidly to changing business process requirements by achieving a very high level of reuse of the standard services.
In the new SOA model the standard services must be managed as products, providing multiple customers a well ordered supply based upon demand forecasting, continuous feature delivery and change management. The cost of service (product) management then forms an ongoing cost that is spread across all users of the service(s).

Summary

SOA can enable an enterprise to create an agile, evolving business environment. However today’s typical implementation activity is not likely to deliver on this objective.

Unlike previous technology trends SOA requires genuine architectural effort to establish enterprise specific solutions that can evolve existing systems into a responsive, cost effective environment. This architectural effort must develop architectural solutions that make agility an inherent part of the systems environment based on deep understanding of how the enterprise is likely to undergo change, and then to establish policy that defines how the enterprise can maintain that agility even while constantly evolving.

This requires a level of formality around architectural policy that is not typically present in enterprises today as well as formality of governance practices to ensure the ongoing agility is not compromised. This requires engineering practices that are specifically designed for the service delivery and management model. The right level of formality must be based a formal, automated service oriented meta model and process to support the provisioning and use of software service assets that ensures conformance to core principles articulated as policies. Minor alterations to existing practices will not merely be sub-optimal, they will deeply compromise architectural and business goals.
It is a business and IT management responsibility to ensure that appropriate organizational agility is embedded in the business processes and supporting information systems. If there is no formal agreement and management of architectural style and principles then SOA investment is almost certainly a waste of money. If the organization’s processes are not suitably reengineered to deliver and maintain the SOA then there will be business as usual.

Consistently high levels of business agility will not happen by accident, it will only happen by design and this requires a step change in the execution of architecture and engineering.

1. Aberdeen Group Survey: Nine of every 10 companies are adopting or have adopted service-oriented architectures and will exit 2006 with SOA planning, design and programming experience." The report indicates a growing and widespread acceptance of SOA technology, especially among large enterprises. SOA is broadly seen as a real technology step forward, with the largest companies, who have the biggest integration problems, leading the way."
   
   According to a Yankee Group survey, 84 percent of companies are already in the midst of a service-oriented architecture project or will undertake one within the next year. So far, most who have already implemented SOA are using it for customer-facing Web sites or portals (30 percent) and help desk and customer support applications (29 percent).
   
   According to a Forrester Survey April 2006 Service-oriented architecture (SOA) adoption continues to be strong, especially for the largest enterprises. Satisfaction with SOA runs very high: Nearly 70% of SOA users say they will increase their use of SOA, and 46% of large enterprise users of SOA use it for strategic business transformation.
2. Enabling the Business Service Network
3. Charles Schwab Responds to Market Conditions and Customer Needs
4. Enterprise Service Bus, refer CBDI Forum report, April 2006
5. The Zachman Institute
6. CBDI Report: Enterprise Framework for SOA
7. CBDI Report - The Service Lifecycle

  Images used in CBDI Journal July/August in Powerpoint format, (Gold subscribers only)

  CBDI Journal July/August in Adobe PDF format (Gold subscribers only)

Reports in July/August Journal

• Main Menu
• Editorial
• Service Architecture and Engineering
• The Role of Business Process Models in Service Planning
• Service Portfolio Planning and Legacy Modernization

Print this page