IMS to DB2 migration is on the radar of many mainframe IT managers. This article explores the issues surrounding the migration of IMS-based data, from the underlying business drivers to evaluating the possible approaches. It also offers a preferred approach that reduces risk and time-to-market while protecting investments.

Background

Modernization has become a central theme in the IBM mainframe world. Even today, 20 years after the introduction of client/server computing and a decade since the introduction of Internet-based computing, the mainframe continues to play a key role in supporting critical processes. But with decades of investment comes islands of data and process logic, often implemented in different generations of technology. It’s not uncommon in most mainframe data centers to find multiple database management systems, multiple development languages, a combination of home-grown and packaged applications, and several architectural styles, including host-based, client/server and Internet-based computing models. These silos create a major challenge to any modernization initiative.

Numerous studies have shown that as much as 80 percent of a typical IT budget is spent on supporting the current software portfolio, leaving precious little for supporting new strategic business initiatives. In this context, modern ization can mean the elimination of duplication, which can reduce costs in areas such as licensing, maintenance, and operations. It also can mean providing an architecture into which new functionality can be rapidly introduced by leveraging existing investments.

For many mainframe pundits, modernization means replacing legacy applications with modern integrated packages. While this may be the most appropriate approach in some instances, the mainframe applications that survived Y2K scrutiny did so because they provide unique value to the business. Discarding this investment in favor of new technologies may resonate well with our consumer-orientated thinking, but it fails to take into account the complexity and risk involved in large-scale IT migration projects and the fact that those mainframe applications contain the logic that defines how organizations work, something that won’t come “out of the box” with new applications. A better way to think about modernization is to use the most appropriate approaches and techniques to bring key applications and data stores into the 21st century while preserving the unique intellectual property that supports the business.

Modernization initiatives focus on several architectural elements, including the user interface, application and process logic, underlying data layer and scalability of the overall design needed to deliver the required service characteristics. Architectural change is often needed in one or more layers to provide support for processes that increasingly span divisions, organizations and channels. The challenge is to minimize risk and cost by focusing on the right layers to address the requirements at hand.

We’ll focus here on the data layer because it can often provide the quickest win with the least disruption to existing application portfolios.

Focusing on the Data

A significant percentage of business data still resides in mainframe data management systems, with much of that mainframe data still in pre-relational data management systems such as IMS. IBM has positioned DB2 to fulfill the central role of data management for today’s modern Service-Oriented Architectures (SOAs). Its rich feature set and ongoing enhancements consistently place it top of the list for flexibility and compatibility with other standards-based data environments. The modernization of the data layer is clear; unlocking the potential of data stored in pre-relational data management systems such as IMS requires that data be moved to DB2. How we achieve that depends greatly on the data in question, the application requirements placed on it, and the rationale driving the need for migration. Let’s look at several scenarios.

Scenario #1: IMS can’t support our rapidly changing reporting requirements. IMS’ hierarchical structure was designed for high-volume transaction processing, not for open data access. So it doesn’t lend itself to flexible reporting in the same way as DB2. If your main challenge with IMS is business intelligence, then there are several approaches available. The most common is data replication. Data is periodically copied from operational IMS databases to a DB2 environment designed for reporting. The copy process often involves complex data aggregation and transformation logic to map transactional records into data schemas designed to support SQL-based reporting tools.

The batch nature of the process means that reporting data is always out of date when compared with operational data stores, which can lead to multiple “versions of the truth.” Some of this can be mitigated by performing ongoing data propagation. Updates to your IMS databases can be synchronously or asynchronously propagated to the equivalent data in DB2. The main disadvantages are the overhead of maintaining two copies of the data, and the additional layer of complexity introduced into your environment.

An alternative and sometimes complementary approach involves the use of gateway technology to gain direct access to IMS data. Gateways are adaptors that hide the complexity of the underlying data structure while exposing a SQL interface for use by applications or reporting tools. While simplifying life for the user, they do require mappings under the cover, which means work for DBAs or system programmers in addition to license and maintenance costs for the gateway software itself.

A third option to address changing reporting requirements is to permanently migrate IMS data to DB2. This allows direct access from reporting tools for timely operational reporting and can greatly simplify the building of data marts and data warehouses. However, this approach has significant implications for existing applications.

Scenario #2: Existing IMS data is needed to support new applications. These new applications may have different characteristics than the original IMS applications. For example, they may have continuous data availability requirements as in Web-based, self-service applications. They may need to support concurrent online users and batch applications or provide a single view of a customer by aggregating information contained across several relational databases, some running on non-mainframe platforms. While these requirements are easy to satisfy using DB2, the process with IMS isn’t as straightforward, both in creating the glue in the first place and in supporting it over time. Application integration technology can ease this complexity somewhat but the simplest way to address the problem is by migrating IMS data to a common DB2 data store.

Scenario #3: IMS is becoming too expensive. This is a frequently heard argument. IMS license costs are expensive and while they can be justified across a large number of databases, they become difficult to defend as more data is placed under the control of DB2. In addition, many IMS shops run third-party IMS management tools that represent a recurring expense above and beyond the IMS licenses themselves. Add to this the fact that IMS DBA skills are expensive and in short supply and it’s possible to build a solid business case for moving from IMS to DB2 based on cost savings alone. Permanent IMS- to-DB2 migration is the only way to eliminate these costs.

So IMS-to-DB2 migration offers a viable approach to address our reporting requirements, is a preferred approach to open up IMS data to next-generation applications, and is the only approach to eliminate the high costs associated with running IMS, short of throwing away our legacy applications.

Application Modification vs. Transparency

There are two standard approaches to undertake an IMS-to-DB2 conversion. The first involves a combination of data transformation and application coding changes while the second involves data transformation and database transparency, leaving the applications themselves unchanged.

Let’s look at what both approaches have in common—IMS-to-DB2 data transformation. IMS data structures consist of keyed fields followed by data items that are “unknown” to the database, but instead are validated and processed in the applications. DB2, on the other hand, locates and joins data records by data value and enforces constraints on the data that can be entered in a field. While this distinction is unimportant to existing IMS applications, it can be critical to both new applications and SQL-based reporting tools. The degree of data cleansing and data normalization required will be directly proportional to the future use of the data in question. If data will only be used by existing applications, the simplest DB2 table structure would be a field containing the segment key followed by a text field containing the remaining data items in the segment. At the other extreme, for flexible reporting, the fields in a single IMS segment could find their way into several fully normalized tables with significant data cleansing work carried out before the migration process occurs. In typical migration projects, we find something in between.

There are potential benefits from the data re-engineering process. For example, we’re able to convert the many different data field representations found in different IMS databases into DB2 DATE columns and clean up date values such as “ASAP,” which are allowed in IMS but not in DB2. We also can universally enforce constraints on other corporate data items that serve multiple applications. Organizations undergoing this initial analysis phase will find it useful to use tools that provide data analysis and data mapping.

Once the data migration details have been decided, the subsequent migration steps will depend on the approach taken at the application end—application modification or transparency.

With an application modification approach, current IMS metadata definitions (copybooks) must be replaced by DB2 table definitions in all applications currently accessing the IMS database(s) being migrated. In addition, each IMS call across all applications must be replaced by a corresponding DB2 SQL call. For example, a Get Unique (GU) call for a root segment using a qualified Segment Search Argument (SSA) could easily be replaced by a singleton select SQL call predicated on the primary key. Given the navigational nature of IMS, this is rarely a one-to-one replacement and often results in wider programming changes. If the GU call is followed by a Get Next in Parent (GNP) loop, multiple IMS segment types at multiple database levels are accessed. The equivalent calls in SQL require complex logic to ensure the correct data is returned in the correct sequence.

Care must be taken to structure the SQL call sequence in a way that doesn’t adversely affect performance. These changes can be partially automated using search and replace conversion programs, but regardless of how the changes occur, there’ll be significant testing required for each application changed. The automation has to parse the existing IMS calls, and be sensitive to inter-related calls, and calls using the many IMS command codes. The automation is further complicated if you use the command-level and the call-level IMS interfaces.

All applications that reference the migrated IMS database(s) must be modified and tested before data migration occurs. The complexity and therefore risk of the project will be directly proportional to the number of applications involved. Usually, strong project management is essential to ensure a successful outcome. The length of the service outage during the actual migration depends on the size of the largest IMS database being migrated. To reduce this outage, pre-cutover work may be required to extract data to separate history databases. This approach will require the addition of programs or programming logic to access these history databases.

There are two different approaches to transparency solutions. One approach is to automatically convert the IMS calls in your application programs to SQL calls. This is a more structured approach to the previous scenario, supported by commercially available tools. All application programs that issue IMS calls are automatically changed. In the second approach, IMS applications remain unchanged. They continue to issue IMS calls, process resulting IMS data, and execute conditional logic based on IMS return codes. The transparency layer becomes responsible for providing a bridge between the IMS programs and the DB2 and IMS databases that hold the production data. To do this, it must be able to intercept the IMS calls and determine the location of the data being requested. Unlike the application modification approach, IMS databases can be migrated gradually, one segment at a time. This can greatly reduce project risk as well as reduce disruption to the business. The transparency solution keeps track of data migrated to DB2 and data still in IMS and, at call time, either allows IMS logic to continue or initiates a DB2 call process. In this process, the IMS call is replaced by a highly optimized SQL call sequence. The retrieved data is then transformed and parsed to re-create the original IMS record and is passed back to the application with the appropriate IMS return code.

There are several advantages to the second transparency approach. Because applications remain unchanged (just re-linked to point to a new IMS call stub), application conversion work is eliminated and testing is greatly reduced. If the long-term objective is to modify the applications, this can be undertaken on a gradual basis after the data is migrated. So we have the ability to divide both data migration and application modification activities into smaller, more manageable pieces. And if, for whatever reason, problems occur during cutover, fallback using a transparency solution is simply a matter of setting a switch to point to the original databases.

This migration approach has been successfully adopted by leading companies in a variety of different industries both in the U.S. and across Europe. A major European bank, for example, unlocked its customer data to enable several new interactive channels for customers, initially with no changes to existing programs. Over the course of several years, they modified programs in a phased approach to natively access DB2 and now no longer use the transparency technology.

Conclusion

A modernization strategy can accommodate legacy applications and data, including data in pre-relational databases such as IMS. We’ve looked at IMS-to-DB2 migration from several perspectives and put forward a preferred approach that has been successfully used by many organizations. The approach that’s right for your organization depends on the reasons for pursuing a migration and the long-term requirements placed on the migrated data.