For the last three years now, we’ve done primary research in the mainframe market to understand where things are headed for the IBM System z platform. This has taken the form of a survey that touches approximately 1,000 respondents from some of the largest, most recognizable mainframe users around the world.

Each year, we pay particular attention to this question: “For every dollar spent on your mainframe operations, how much of it is allocated to software, hardware, labor, and other (e.g., facilities, energy, etc.)?” Our respondents report that software is always the largest line item and also one of the largest growth areas (in contrast with distributed environments, where labor is consistently reported as the biggest cost). With software costs large and visible, they become the focal point for the inevitable cost-cutting hammers that finance guys (with their spiffy spreadsheets and limited understanding of the underlying story) like to wield.

This raises the question, “Is mainframe software too expensive?” We’d argue that no, it isn’t. Moreover, shifting more of the budget into software from the hardware and labor categories is likely to result in even better overall Total Cost of Ownership (TCO) for large, compute-intensive data centers.

Before we argue about the budget mix, however, we must first dispose of the argument that “the mainframe itself is too expensive.” We must argue that, for a given workload, the mainframe is a better alternative than solutions based on other platforms for a given workload.

Let’s assume the value of the workload is constant and the business has justified deploying the application. The only issue in question is whether the application portfolio is most cost-effectively hosted on a mainframe or in a distributed systems environment.

To open our assault on blatant falsehoods concerning the “unreasonable expense” of the mainframe platform, here’s a quote from Gerry Shacter in PC Week from 1992:

“Mainframes are big for one reason. It was too hard to manage a lot of small ones. PCs will get bigger, too, until there are no more because the one that’s left will be so big, it will be able to handle all the work. It will be called a mainframe, and these upstart programmers will think they invented it.

“One day, the management of the Fortune 500 will wake up and look upon its thousands of PCs and wonder what they are all doing, finally realizing that it would be much easier to replace them all with one mainframe and then wonder what it’s doing.”

Within these rather amusing predictions, we find the truth. First, that for large, resource-intensive workloads, there’s a concept known as “economies of scale,” and second, the labor to manage many little things attempting to work together is much higher than the labor to manage one big thing doing a lot more work from a centrally managed perspective.

While it’s unreasonable to assert that all low-end systems will eventually morph into one giant mainframe (despite attempts by folks such as Amazon to create a monolithic “cloud”), we’re on pretty safe ground by saying that the case is quite clear for data centers running massive databases, with high transaction rates, mixed workloads, and extreme security and availability requirements (these include banks, major corporations, etc.). The annual mainframe survey referenced earlier shows these are exactly the data centers that are growing mainframe capacity and adding new workloads, thus driving larger and larger configurations.

Given that banks and other financially motivated enterprises like to collect money rather than spend it, why would they not only continue to spend and invest in their existing platforms, but actually expand their investments? Perhaps they’re good at math and have deduced that they get more work done for less money by leveraging mainframe technologies.

Wayne Kernochan points out in a 2006 Illuminati report, that to correctly frame the mainframe vs. distributed TCO problem you must consider real apples-to-apples configurations. This means looking at portfolios of 10, 20, or even 50 applications running in a single environment vs. trying to measure individual apps running on standalone servers or even blade/grid configurations. Kernochan gets to the point when he says, “System z wins on the basis of administrative costs. … The number of administrators required to run … distributed systems increases nearly linearly …, while a single mainframe’s administrative costs barely increase at all.”

If we assert that for a large enough environment, the hardware and software costs tend to wash in the TCO battle, that leaves a radically diverging expense line for labor to settle the question, and it settles it in favor of System z.

But there are other issues. In an early 2007 research report, the Butler Group cited similar findings and specifically highlighted other reasons for the TCO advantage.

Quoting the report, Stephen Swoyer wrote in Enterprise Systems Journal, “What’s more, Butler Group maintains, the mainframe’s integrated power and cooling capabilities, along with its small footprint, appeal to large customers who are attracted by its power-saving features and reduced floor space requirements. Butler Group officials also highlighted System z’s virtualization capabilities, which—thanks to the mainframe’s best-in-class virtualization support (via z/VM)— allow it to host thousands of application or operating system images.”

A discussion with an IBM representative provides several interesting points that also weigh into any TCO discussion:

• Mainframes tend to run heavily utilized. For a mainframe system to be running at 80 percent or higher utilization isn’t unusual. In a distributed environment, numbers as low as 20 percent are common. While it would be difficult to compute, one can easily see massive waste in energy and management resources to run systems that are basically just generating heat!

• While mainframe software costs are quite visible, the equivalent numbers in the distributed world are hard to come by. The lack of infrastructure for calculating costs and charge-backs makes any true analysis difficult. It’s likely people are spending much more than they realize. The mainframe market has responded with unique, innovative programs and tools to let customers manage software costs and enjoy a more controlled software spending environment.

• The simplicity of the mainframe environment makes such mundane issues as disaster recovery much more practical, while the same task in a complex distributed environment is much more expensive and is often abandoned in frustration.

For a case that illustrates some of the less obvious things that can come up when considering the platform cost question, see the recent Forbes.com article, “Servers: Why Thrifty Isn’t Nifty” by Kenneth Brill (www.forbes.com/technology/2008/08/10/cio-cheap-serverstech-cio-cx_kb_0811servers.html). He points out that the deployment of all the new blade servers is driving a massive increase in data center costs, and in particular, companies are wildly underestimating how much power these things consume. For example, “One company’s IT department decided to invest $22 million in blade servers but forgot to inform facilities. The facility investment required to merely plug in the blades was an unplanned $54 million. An additional unplanned $30 million was required to run the blades over three years. So what appeared to be a $22 million decision was really an enterprise decision of over $106 million.”

While every pundit and platform bigot will assert their product or approach will yield the best result, there’s more than enough evidence and supporting rationale to claim that for the large, computationally and transactionally intense, mixed application workloads found in a typical, information- driven enterprise, the mainframe is a cost-effective, superior platform in terms of technical performance and overall productivity. If this weren’t the case, IT professionals in those exact environments wouldn’t be investing in the platform for precisely those purposes.  Based on these considerations, we will now assert that for workloads of the scale and importance typically found on the mainframe, the overall TCO is demonstrably superior. Let’s now move to break down and evaluate the components of that TCO. Survey results show that respondents allocated their mainframe operations dollar to software, hardware, labor, and facilities/other. The answers were 40 percent, 26 percent, 27 percent and 12 percent, respectively.

This stands in stark contrast to distributed environments where the allocations have labor running at 43 percent and software at 27 percent, essentially flipping the result we found in a mainframe- focused context.

Kernochan cites Infostructure Associates studies that suggest that, “Simplicity lowers administration costs. This simplicity includes both automating user administrative tasks and reducing complexity of the application and enterprise architecture.”

The same studies note “administration costs, including deployment, upgrade and maintenance—and especially database administration costs— are the dominant components of TCO, and their percentage of overall costs continues to increase.”

So if one can argue that the basic hardware and software found in distributed computing environments is nominally less expensive than the mainframe equivalents, but we observe that for large, complex enterprise configurations, the mainframe environment offers a superior TCO, then we must be getting more work out of the equivalent mainframe resources than from an alternative distributed- based solution. The excessive labor component of the distributed solution is eliminating any initial cost advantage based on component expense.

Consider these points:

• The mainframe provides a much simpler overall architecture and operating environment for running multiple-application, mixed-workload environments in a highly reliable, secure fashion.

• The mainframe’s inherent advantages for security, virtualization, and disaster recovery eliminate a host of complexity encountered when attempting to replicate such features from a grab bag of distributed systems components.

• The amount of labor required to manage systems of similar size and capacity is much less in the mainframe context than in a similar distributed configuration.

In our survey, the primary reasons respondents cited for deploying the mainframe were availability, transaction performance, security, and reliability. These characteristics are proven attributes for the mainframe; achieving these same levels of performance with distributed components would require a complex, carefully planned design if it could even be achieved at all. The result is a simpler, more efficient architecture.

While some labor advantages in the mainframe arena are explained by the simpler overall architecture and more standardized environment, some of those advantages could be offset by the scarcity and costs of mainframe skills. The industry is keenly aware of the aging mainframe workforce. But our survey respondents still report lower labor costs, on average, than the general data center.

Meanwhile, management software and automation tools are making a difference. As Kernochan points out, simplicity is driven largely by automation, and automation allows a staff to manage larger workloads more effectively.

While discussing IMS staffing concerns with a customer, the customer noted that the reason they were able to maintain a data administration staff of only three people over the last 12 years was based on their automated tools. He noted that his database volumes had increased more than 100 times in that period, that his processing requirements were vastly more complex, yet he hadn’t added any staff in that entire period for IMS data administration. They had more than 45 production databases and more than 100 test/development environments, yet no increase in staff.

So it’s clear that investing in software to automate administrative tasks will drive down overall labor requirements, while shifting that expense into the software category. But this same labor advantage is a critical element in driving the overall TCO for the mainframe (assuming sufficient size and scale of the workload) lower than distributed alternatives.

So we leave the reader with the question: “Is mainframe software too expensive?” If by spending more on software and less on labor we achieve a more cost-efficient environment that happens to be more reliable, secure, and efficient, then one wonders, “Why aren’t we spending more on mainframe software?”