Chapter 3. SOA Integration — Functional View, Implementation, and Architecture
Let us not expect this chapter to be a primer on Services-Orientated Architecture (SOA). The focus of this chapter is to place SOA in the context of Modernization. SOA can be over hyped and has been, in some cases, exploited by IT vendors as the 'holy grail' of software architecture. However, in the context of Legacy Modernizaton, an SOA Integration architecture can bring your legacy environment into the world of the World Wide Web, Web 2.0, and all the other latest Internet-based IT architectures. Within days, a legacy system can be accessed via a web browser. This is one of the biggest advantages Legacy SOA Integration has over other types of Legacy Modernization. Your time-to-market in weeks, instead of months or years.
Although not a primer, let's first make sure we are all talking the same language when we talk SOA. Though the concepts behind SOA are not new, SOA is also not yet mature. SOA in it current form has really been around only five years now. The concepts of standards-based protocol handlers, pre-defined communication schemas, and remote method invocations have been around for decades. To get a better understanding of SOA, let's see what SOA is, and what SOA is not:
Legacy SOA Integration is one aspect of an IT architecture called Enterprise Information Integration (EII) and Enterprise Application Integration (EAI). EII is a more data-centric approach to information integration, and EAI is a process or application approach to information integration. EII and EAI include data consolidation, data federation, shared data access, and remote method invocation:
- Consolidation — Data consolidation moves all disparate data sources into one central database. This is akin to a data warehouse, and in many ways it can use the same approaches and tools, but with update, insert, and delete capabilities. In many cases, the first stage of data consolidation involves creating a data warehouse, as migrating all the existing applications that access the data can take years or even decades.
- Federation — Data federation leaves the data in the individual data source where it is normally maintained and updated, and consolidates it on the fly as needed. In the case of Legacy Modernization, multiple data sources, many of which are non-relational, will appear to be integrated into a single virtual database. The platform will mask the number and different kinds of databases behind the consolidated view.
- Shared Data — Shared data integration actually moves data and events from one or more source databases to a consolidated resource, or queue, created to serve one or more new applications. Data can be maintained and exchanged using technologies such as messaging queues and ESBs.
- Remote method invocation — The first three options focused on the data side, or EII. The last option is focused on the processes and business logic, or EAI. This is integration at the application layer instead of the data layer.
SOA Integration is a combined data federation and remote method innvocation approach to solve EII/EAI in a Legacy Modernization project. Traditionally, EII has been achieved using Extract, Load, and Transformation (ETL) products. With the advent of SOA technologies such as ESBs, more customers are willing to use ESB products for information integration. ETL vendors have not been standing still and have introduced Web services, and standards-based transformation and workflow engines instead of proprietary software. This means that the differences between ETL and ESB technologies are blurring. In addition, the capabilities of Business Intelligence (BI) and Business Activity Monitoring (BAM) technologies are converging. BI tools are becoming more real time, and BAM technologies are offering reporting capabilities.
A major component of an EII/EAI infrastructure is Master Data Management (MDM). MDM is a single version-of-truth for your information, which is maintained in a central repository. MDM is all about sharing, reusing, and enabling maximum interoperability of your core master reference data. This allows your applications to access the data through a common object or model and a common set of access schemas. Therefore, it does not matter where that data might reside. Your application can access the data through a virtualized access layer. As you can see, MDM is very important in an SOA Integration environment. MDM is a large topic and a lot has been written in this area. Therefore, we are not going cover MDM in this book.
Now we know what Legacy SOA Integration is, and why we are going to do it. We must now embark on the journey of how we are going to get from a monolithic mainframe to a SOA-enabled mainframe. Before we can just start calling our mainframe or midrange systems, we must first understand the following from a technical perspective:
- What does our business community really want to see?
- How are they going to see it?
- When and where do they want to get the information?
- What should my first project be?
- What will our architecture look like?
We will answer all these questions in this chapter.
The fundamental question many business people ask today is, "Why can't enterprise IT architectures be as straight forward and easy to use as Google, MySpace and other web sites?". This is a valid question, to which the answer is, "IT systems don't have to be and should not be".
As any good IT architect knows, you must balance the combination of cool products, time-to-market, maintainability, ease of use, and cost of the technology. We have not even begun to build the architecture and we are talking of costs for a very good reason. Cost, as in total cost of ownership (TCO), is something every IT decision maker must consider. TCO is not just the upfront cost of the product, but also the cost to maintain, enhance, and manage the environment.
Just placing Web services on an already unwieldy mainfame architecture that does not scale when interfacing to other systems is not the answer. The SOA Integration architecture must be fully secure (in terms of access and data), capable of replication, scalable, and performant. The architecture must also take into account the legacy system that will need to be both a source, and a consumer of Web services.
Unfortunately, most companies with systems older then ten years are starting with an architecture that is appropriately called the Accidental Architecture. This architecture probably looks something like this:
Note
These lines should cross, go every way, and make a complete mess. The accidental architecture diagram above shows a simplistic view of the integration points in the mainframe systems. Most mainframe systems have far more complex infrastructures with more database vendors, more hardware systems, and systems communicating with each other. As the real diagram would most likely be complex to follow, we have used a simpler version here.
The idea in modern SOA architectures is to eliminate this complex IT architecture. Back in the old days, business IT infrastructures were rigid and complex because memory was expensive, the mainframe disk was small, few people knew about technology, not many technologies or vendors existed, and most IT infrastructures had to be custom coded.
The other thing to consider when developing a Legacy SOA Integration Architecture is that in most cases business processes and workflow are likely to change. Changed business processes entail mapping business services, while taking into account the current business flows, to the new business use cases and creating a business process workflow to implement these use cases.
SOA cannot be discussed without discussing the topic of accessing internal and external Web services. When Web services and SOA were first conceived, the grand idea of developers and users dynamically discovering and using Web services from all over the Internet was at the forefront of the discussion. In reality, this idea was way ahead of its time. Although, almost all SOA implementations leverage internal services, they do so on a very limited basis. Therefore, this chapter will focus on the internal legacy services.
According to the Aberdeen Group: "Organizations that are SOA-enabling their legacy applications on the legacy platform are outperforming those that are using any other approach. They report better productivity, higher agility, and lower costs for legacy integration projects". This is one IT analyst's perspective but it does highlight the importance of SOA integration in any company's legacy modernization strategy.
Note
Keeping It Real: The most difficult aspect of building a new architecture that is completely SOA-enabled is the vast amount of business logic that exists on the mainframe, mostly in COBOL code. A Legacy SOA Integration architecture preserves that legacy business logic but incorporates it into the new world of relational databases, SOA, and Web 2.0.