zJournal interviews Steve Womer, a senior consulting IT Architect at Nationwide Insurance, who has more than 27 years of experience in mainframe and distributed systems. For the past five years, Steve has been designing and implementing Linux solutions across various platforms.

z/Journal: So, tell me, Steve, what problems were you trying to solve?

Steve Womer: Nationwide has a large number of distributed servers. Most of the servers have a peak utilization of less than 50 percent, with an average utilization between 5 and 15 percent. Our server provisioning time was measured in months to acquire and configure a new server. Dynamically allocating processing power was difficult, for example, whenever a given application needs more processing power on Mondays, or during the end of month. For these reasons, we typically over-provisioned the servers in our distributed world. We also were facing a data center power and floor space scarcity, and potentially tens of millions of dollars investment to upgrade. Finally, Nationwide was looking for an opportunity to dramatically reduce the TCO [Total Cost of Ownership] of our distributed server and Web hosting environment.

z/J: Please tell our readers what made mainframe Linux attractive to you.

SW: Nationwide could address all our identified issues with a mainframe Linux solution. The mainframe’s ability to share resources across multiple applications and workloads was very appealing. The concept of sharing resources and virtualization is welltested in the mainframe world. The hypervisor software (z/VM) is more than 40 years old and Nationwide already has deep experience with it. The pricing model for running the OS and middleware on zSeries IFLs [specialty mainframe processors designed to run Linux workloads] is significantly different from traditional z/OS or distributed platforms.

z/J: Did you have prior experience running Linux on the mainframe?

SW: Yes, Nationwide had previously tested running some simple applications on MF Linux. In addition, before I came to work at Nationwide, I had about three years of experience in designing Linux solutions in another mainframe environment.

z/J: Please tell us more about the technology building blocks involved.

SW: What did we build? We used two z/990 servers with z/VM 5.1 as the hypervisor. We currently have both Red Hat Linux AS 3.0 and SuSe Enterprise Linux 9 deployed in a three-tiered architecture, leveraging WebSphere 5.1, MQ, and Portal Server. We’ve built a redundant, highly available environment, complete with a replicated disaster recovery solution for our production environment. The environment was mapped to our J2EE [Java 2 Enterprise Edition] reference design, as shown in Figure 1, where IHS is the IBM http server, WAS is WebSphere, and DB at this point is DB2 UDB 8.

z/J: Did mainframe Linux deliver value and perform as expected?

SW: Yes, primary Web applications run on zLinux today. Nationwide has five mission-critical applications already in production, with 10 more in test mode. Demand for the environment was much greater than we anticipated. We originally estimated that, by December 2005, there would be 150 virtual servers in the Nationwide environment; there were actually 250. During testing of our ability to scale Nationwide’s Web domain, we load tested Nationwide.com to 22 times its anticipated peak, while simultaneously loading another major application to five times its peak usage, without exceeding our service level agreements.

z/J: What were the main results you were targeting?

SW: With the zLinux virtualization project, Nationwide was targeting a lower TCO for our servers used in our J2EE applications. In addition, our requirements included creating a highly available environment that could be quickly provisioned. The new environment would support both development and production. Nationwide also was looking to leverage the J2EE standards recently accepted by Nationwide’s architecture review board. The zLinux virtual servers also must include a disaster plan congruent with our overall continuity management policies.

z/J: How did you measure results?

SW: We measured our results with a follow-up TCO analysis and meeting service level agreements with our users.

z/J: What key factors contributed to your overall success?

SW: Nationwide’s zLinux project is currently estimated to save $16 million dollars over the next three years, not including floor space. We also were able to provide a reduction in server cost of more than 50 percent to our customers. The zLinux system saved significant data center floor space and power consumption. We estimate an 80 percent reduction in floor space, as compared to an equivalent number of distributed servers. Nationwide support requirements also were reduced; we require 50 percent fewer FTEs [Full-Time Employees] to support hardware and OS due to shared libraries in zLinux. Our provisioning speed was reduced from months to days. Nationwide also created a new disaster recovery process to support the zLinux virtual servers. This process reduced our recovery point objective from days to one minute and our recovery time objective from days to hours.

z/J: That’s very impressive, congratulations! So, tell me, what valuable lessons were learned during deployment?

SW: Nationwide built a business case first and obtained executive sponsorship. Then we assembled a dedicated team to finalize the design and implement the infrastructure. We also developed a rate structure, and created our operational process, all within four months. Nationwide then went back and reviewed our business case, and validated our original business case to the results.

Nationwide can achieve dramatic changes with a focused, dedicated team. The TCO analysis was critical for justifying the project, but you need to include everything. A TCO with only hardware and Linux OS doesn’t show the full benefits. Adding capacity is easy; start small. When creating servers, don’t create virtual copies of distributed servers; leverage the architecture. Nationwide took the time to determine what resources our applications needed—not what was provisioned in a distributed world. Significant adoption requires a good economic incentive for application teams.

In our zLinux project, the sustained support of senior management also was invaluable. Good enforcement and technology governance of standards were critical to Nationwide’s success.

We found Unix skills were easily transferable from dedicated environments to zLinux. Nationwide is in the enviable position of having extremely deep z/VM and zSeries hardware skills. We could not have been successful without these individuals. Finally, don’t underestimate the mental shift application teams will face. Nationwide spent many hours explaining and translating the zLinux environment to our distributed application groups.

z/J: Please tell our readers more about how you built the business case. Was mainframe Linux a hard sell?

SW: Management buy-in was relatively easy once we presented the business case. It’s important to move the decision from a technology to a business decision. We first constructed as detailed a TCO as we could on both the existing environment and our proposed new environment. This included documenting all our assumptions for the virtual environment. The keys were as follows:

• In the shared environment we would be using about 20 percent of the floor space and power consumption we would use if we continue to deploy stand-alone servers and racks.

• We had already demonstrated in an existing z/VM environment how we could quickly deploy and re-deploy servers. The only way to reduce our provisioning time in the stand-alone environment was to pre-purchase and stage our servers in anticipation of requests. This would mean sunk costs that we may or may not recover. In the virtual environment on a z/990 we had the ability to start relatively small and grow the environment as we needed more resources.

• For the zLinux environment, we projected that our FTE support costs would be 65:1 as opposed to our current support level. Since we had total requests for 100 servers, we compared the current and go forward total costs in both stand-alone and virtual environments. Over a threeyear period, our virtual environment TCO was approximately one-half the cost of the traditional rack deployments.

• We then followed the business case with a proof of concept, using our existing z/VM environment to validate that our applications could receive the same SLAs as in the existing distributed environment. We were able to demonstrate that a particular application in need of expanded resources could scale to five times its current peak requirements in the virtual environment on demand.

z/J: Did any platform turf wars erupt? Where did you encounter resistance along the way?

SW: Most of our resistance came from the existing distributed groups. Their resistance was based on their lack of understanding in what the zLinux environment could give them. We addressed this resistance during our proof-of-concept by demonstrating similarities in the distributed environment and what new features they could leverage in zLinux.

z/J: Has this success with Linux changed your vision?

SW: Yes, but not so much on Linux. The success of this undertaking has reinforced the importance of making an environment operational. You can create the best technical solution in the world, but until you can make it operational, it will be of little business value. This requires standards, an updated process to change, and deployment. Nationwide had the foresight to drive a technical solution without losing focus on the operational side.

z/J: What kind of future do you see for Linux on the mainframe?

SW: I see a bright future. By merging mainframe technologies with distributed technologies, we will have the best of both worlds.

z/J: Please elaborate.

SW: The zLinux environment will only continue to grow and evolve. The stability of the hardware, combined with the lower cost of ownership and speed to market of new features, will bring many new features to the environment. The zLinux environment at Nationwide has proved to be flexible, responsive, and cost-effective; these are the messages that will continue to drive virtual servers on a zLinux environment.   

Steve Womer is currently a senior architect at Nationwide Insurance in Columbus, OH. He has more than 27 years of experience in mainframe and distributed systems. For the past five years, he has been designing and implementing Linux solutions across various platforms. He participates in COMUG, a mainframe users group in central Ohio. He also enjoys public speaking and has presented at Linux World and several open forum events. He is a former consultant and open source evangelist.