As large organizations routinely ask CIOs to deliver real-time business data to executives, partners and customers, CIOs are turning toward Service-Oriented Architectures (SOAs) to help make that happen. Because much of their core data resides on mainframe computers, they’re asking: “What’s the best approach to enable the integration of legacy data and functionality into an SOA?”
Enabling legacy modernization means surpassing the point-to-point integration approach, which isn’t open standards-based. Using Web Services to transform legacy data and functionality creates valuable assets that Java, .NET, and other application platforms can consume and reuse.
How do you “modernize” legacy assets to let them integrate with the larger enterprise into an SOA while also capitalizing on the speed, capacity, reliability, and security that make the mainframe so valuable? The first step is determining which ones are worth moving forward.
Tolerate, Integrate, Migrate or Eliminate?
You need to evaluate each legacy asset on its own merits, assigning it to one of four categories:
- Tolerate: These are fine as is. They’re self-contained, used on a limited basis, and tend to be isolated from the rest of the organization. The best approach is to maintain and manage them without making any changes.
- Integrate: These assets are at the core of business, carrying out complex business processes. They’re highly efficient and stable. Replacing them with new, unproven systems is too risky. Keep them on the mainframe and integrate them with the larger enterprise.
- Migrate: These assets are no longer meeting your organization’s demands. They’re expensive to operate, require specialized skills that are in short supply, and were often developed by vendors that are no longer viable. It’s best to migrate them to another platform.
- Eliminate: These assets aren’t needed anymore. They’ve been rendered redundant by other applications based on standardized technology. Eliminate them.
For applications you want to integrate, consider the basic approaches to exposing legacy assets in an SOA:
- Service-enabling the database
- Web and service enablement through the User Interface (UI).
Each approach has pros and cons, and there are few absolutes, but there are some guidelines you can use to choose the best approach for your environment.
Service-Enabling the Database
The simplest way to expose your mainframe data, at least in concept, is to provide more-or-less direct access to the database through some kind of adapter. This type of access makes particularly good sense for Web Services that serve a reporting function and result in large volumes of data (e.g., many records from a small set of files or tables).
Database service-enablement solutions exist from several vendors. They have greater or lesser complexity and performance, but all have a way to access the data and a way to expose the data as Web Services.
Some ways to access the data include SQL, either natively or via a gateway product, or via direct Application Program Interface (API) calls (for example, IMS DL/I or Adabas Direct Calls). Often, SQL-based access through some kind of gateway product is already in place from a previous modernization project. If your organization has already implemented this kind of access, your data access task might be a lot simpler. Usually, all that remains is to expose the data as Web Services, typically via a fairly generic Open Database Connectivity (ODBC) or Java Database Connectivity (JDBC)-compliant Web Services adapter like you might find built into an Enterprise Service Bus. Even without SQL access in place, there are several solutions available from various vendors that access mainframe database systems via their native call interfaces and expose the data as Web Services.
Unfortunately, service-enabling the database can be problematic for several reasons, many of which will be familiar to you from prior iterations of “mainframe modernization.” Think of opening up SQL access and watching inexperienced users submitting “queries from hell,” for example. You can get into the same kind of difficulty with Web Services, but the good news is that it’s much harder to have it happen by accident.
If service-enabling the database makes the most sense, then the most “natural” way to expose access is through well-defined “data services,” essentially, small, self-contained pieces of code, which act as transactions that encapsulate access to specific pieces of data based on tightly defined parameters. Some Web Service data access products will even provide a wizard that will automatically generate this functionality for you.
There’s a Web Services equivalent of opening up SQL-based access to your mainframe database; it’s called XQuery. XQuery is a powerful, flexible way to provide procedural access to data (and all kinds of other things). But all that power and flexibility comes at a cost. It’s just as easy for an inexperienced or malicious user to write an XQuery program that consumes a tremendous amount of resources as it was for them to write an unconstrained SQL query that did the same thing. Whether it’s a runaway XQuery program or a runaway SQL query, you’ll find yourself getting taken off your IT department’s holiday card list.
Runaway queries aren’t the only potential pitfall, however. You may find, depending on the vintage of the database and the applications that access it, the data in the database is completely opaque to anyone but the application developer. Examples might include compressed application-specific encodings that made sense when DASD was expensive, but don’t today. To avoid propagating these now-obsolete and application-specific conventions throughout your organization, you’ll need to write some code to turn these cryptic codes into a more widely understood representation. This issue is bad enough when you’re providing read-only access; it gets worse when you provide update access and you need to provide data validation for input, too, to ensure what database types usually call entity integrity.
The last example we’ll offer of how database service enablement can be problematic also occurs when you provide update access. This may include how to enforce referential integrity, or how to ensure your users can’t insert orphan data items (detail rows without a master, for example). If your database system doesn’t automatically enforce referential integrity, you’ll need to consider this when you code your data services.
If this is all beginning to sound like it may involve a lot of coding, then it’s time to look at the next style of access.
Service-Enabling the Application
The major benefit of reusing application code to provide data access is that many of the kinds of problems described previously (decoding cryptic data, entity integrity, referential integrity) are already addressed by existing application code. In addition, providing access to application code often provides other benefits such as several more layers of data validation, and even access to entire processes that may be triggered by invoking a transaction.
How it works depends on how the application was coded and what environment it runs in. Usually, it also depends on having access to the application source code. Let’s consider a few examples:
- A COBOL application running under CICS
- A Natural program running under Complete
- An Oracle stored procedure.
CICS programs can act as either Web Service providers or requesters. Depending on the technology used, it may be necessary to write a wrapper program to separate the business logic from the Web Services logic, although a good product will provide a wizard or other assistant to help generate the mapping between SOAP messages and your application data structures.
In the case of Natural, you’ll find there are products that provide the necessary connectivity between SOAP and Natural, even with bi-directional connectivity. As with our COBOL example, your Natural programs can act as either Web Service providers or requesters.
Finally, stored procedures in Oracle’s database products, regardless of Oracle version (Oracle7 and above), can be invoked via ODBC or JDBC connections through the same kinds of ODBC/ JDBC-compatible Web Services migration tools you might use to provide SQL-based access. However, by exploiting stored procedures, you can mitigate most of the concerns that come with opening up unrestricted SQL-based access. Moreover, most modern, Oracle-based applications exploit Oracle’s built-in features that enforce entity and referential integrity.
Compared with service-enabling the database, service-enabling the application may sound like a panacea, but there are circumstances where it doesn’t apply or is less than optimal. For example, when the Web Service is providing a reporting type function, it may be inconvenient or inefficient to, say, invoke a CICS program 500 times to retrieve 500 data items, when the same data could be retrieved with a single SQL statement or even 500 low-level database calls. In some cases, service enablement for SOA may not be possible without access to and knowledge of the application source code.
It’s time to consider service enablement through the UI if: Programmatic service enablement may sound like a panacea, but there are circumstances where it doesn’t apply or is less than optimal.
- Application source is unavailable
- The code can no longer be compiled
- Your organization no longer has the necessary skills to work with the old code
- You’re unwilling to risk destabilizing the code.
Service Enablement Via User Interface
Tools that can exploit the functionality of an application by directly accessing the UI layer have been used for some time for Web enablement. Today, with more advanced application modernization products, you can create Web Services without having to touch the application code—all in a completely non-invasive manner via the UI.
These tools work much the same way a human operator does: by leveraging functionality and workflow through the UI. They read information from fields on the screen, putting information into input fields, and hitting the virtual ENTER key or function key to submit the filled-out panel or form.
A key area of differentiation among tools is in how flexibly they can parse screen content. Low-end tools may take a simplistic approach based on absolute screen positions (row and column positions), whereas more sophisticated tools can recognize field labels and may be able to tolerate minor changes in screen layout without sending you back to the drawing board. Most tools emulate a variety of 3270-type terminals and this further ensures these kinds of tools can access virtually any mainframe application that has a UI, regardless of development language used, transaction monitor it runs under, data store used, whether it’s a package or developed in-house, and even whether it runs in your data center or has been outsourced.
Once you’ve taught the tool to understand your application’s screens, it exposes the functionality through some kind of application interface. Here, we’re most interested in Web Services, but the better tools also offer other approaches to SOA such as integration with Java and Microsoft’s .NET framework.
Although many products in this space are advertised as offering these capabilities without any programming, you should be aware that this approach sometimes leads to compromises in functionality and you may find that only the simplest cases are accommodated without any coding. However, many products incorporate scripting or full-blown coding capabilities so they can handle even the most complex interactions; the best do so using industry-standard languages such as C or Java rather than proprietary scripting languages.
Early implementations of modernization tools that leverage the UI often required the installation of a separate expansion card into a dedicated PC that provided SNA connectivity and hardware terminal emulation. With the combined widespread availability of TCP/IP on the mainframe, powerful PCs and connectivity to the mainframe via the corporate intranet, almost all UI service enablement tools are now implemented entirely in software.
Besides eliminating reliance on costly, proprietary hardware add-ons and a dedicated PC to run the product, today’s service-enabling mainframe modernization tools often include powerful, built-in capabilities to handle complex multi-screen navigation paths through mainframe transactions, including those that use pop-up windows. They also contain sophisticated error-trapping and exception handling functionality for error conditions.
So when would you use a tool for service enablement that leverages the UI? The most extreme cases are the easiest to define. If you can get to the application through a terminal emulator, you can generally get to it via one of these tools, so they provide almost universal connectivity. Many of these tools also let you get up and running quickly. These characteristics make these kinds of tools ideal for use in situations where you need to leverage the business rules built into the application code, but service enablement for SOA is undesirable or impossible because the code must not or cannot be modified. Because the barrier to entry is typically pretty low with these kinds of tools, they’re also complementary to tactical or one-off integration projects.
Tools that offer service enablement through the UI are well-suited to integration tasks that feature a light-to-moderate, transaction-oriented workload—where there’s a need to get up and running quickly, where the application already exposes the necessary functionality through the UI and, of course, when there’s simply no other way to get to the application for technical or political reasons.
However, because they’re limited, by definition, to the kinds of interactions that can be achieved through a terminal interface, they’re not best-suited to inputting or retrieving large volumes of data in batch because of the large number of screen interactions required. Neither are they well-suited to ad hoc data access because they’re bound to the access paths inherent in your application’s screens. If you anticipate high volumes of data being passed back and forth, or have a requirement for ad hoc data access, then the direct data access technologies may make more sense.
Other Considerations
We’ve considered the performance implications that apply to each of the integration technologies and approaches outlined above; let’s now take a brief look at other areas that deserve consideration.
In many ways, availability is the flip side of performance. A tool that provides quick turnaround in response to a large query may be doing so at the cost of large amounts of resources, thus reducing the resources available to users. These resource distribution issues can often be mitigated by careful design, thoughtful choice of appropriate integration technology, and use of standard operating system facilities to govern resource use and execution priority.
The approaches and solutions that are available to address security vary with the different technologies discussed. Ultimately, standards such as WS-Security promise to bring standardized approaches to end-to-end authentication and authorization for common integration scenarios and interoperability between the diverse security infrastructures that exist in the typical enterprise. A key decision is whether to attempt to represent the user identity from end to end (i.e., your mainframe application “sees” the same user identity as the Windows workstation or Web browser they’re logged in to) or whether to build some kind of application proxy (i.e., there’s a piece of code somewhere that connects to your mainframe application via a generic application user ID, which re-implements some of the security mechanisms built into your application). End-to-end identity is arguably more secure and provides better trace-ability and auditing, but is typically harder to get right. Application proxies offer more flexibility and may be the only way to tackle the trickiest security integration issues, but they, too, require their own set of compromises.
Some of these headaches may go away with the widespread adoption of practical security standards, but in the meantime, there are numerous vendors with solutions that address different areas of the security puzzle. Until the various standards are better established and vendors have started to roll out standards-based solutions, the best advice is probably to do as most application integration vendors do: partner with firms that specialize in security solutions for the emerging Web Services-to-mainframe world (such as Vordel, Systinet, or Forum Systems).
In the absence of complete, widely adopted security standards, and especially in the case of home-grown applications or security mechanisms, the need for flexible, end-to-end security solutions may even force your hand in the choice of integration technology. Of the approaches discussed here—service enablement of the database, service enablement of the application, or service enablement through the UI—it’s the last of these that offers the greatest flexibility and represents the only technology that virtually guarantees the ability to integrate with your application’s security. It may give you the best option as a point of control for implementing end-to-end identity.
