You need to put more work through your IBM CICS Transaction Gateway (CICS TG) for z/OS systems but you also need to monitor and audit them to ensure they continue to deliver the Quality of Service (QoS) your growing business requires. How can you do this? Help may be at hand with performance monitoring features in new releases of CICS TG and IBM CICS Performance Analyzer (CICS PA) for z/OS.
This article describes the new statistics feature of CICS TG, discusses enhancements to the CICS PA product that provide support for CICS TG statistical data, and analyzes a real-life scenario to show how to use the new statistics to diagnose and resolve a performance bottleneck.
CICS TG and the New Statistics Infrastructure
CICS TG is a mature mainframe product that’s been available, under various guises, for more than 10 years. It’s widely used to provide remote access to mission-critical transaction processing systems from Java and Java 2 Extended Edition (J2EE) applications.
Before CICS TG V7.0, it was difficult to access real-time information online about CICS TG performance. To address this issue, CICS TG V7.0 introduced 36 statistics and a system monitoring infrastructure that lets you retrieve online statistics by issuing a z/OS system MODIFY command against the Gateway daemon address space, or programmatically via the statistics Application Program Interface (API).
CICS TG V7.1 extended the number of statistics to more than 100. These statistics provide real-time metrics on key operations and the current status of Gateway daemon instances. This opened up the “black box” and made it possible to gain insight into the operations occurring in the CICS TG. CICS TG V7.1 also added support for writing statistics to System Management Facility (SMF) as type 111 records.
CICS PA
CICS PA is a powerful offline reporting tool that helps programmers and administrators analyze the performance of their CICS systems. CICS PA produces reports and extracts from SMF records in sequential data sets created by the SMF dump program, IFASMFDP. CICS PA interprets SMF records written by:
• IBM CICS Transaction Server (CICS TS) for z/OS (type 110 records)
• System Logger (88)
• IBM DB2 and IBM WebSphere MQ accounting (101 and 116, respectively)
• IBM Tivoli OMEGAMON XE for CICS (112).
A recent update to CICS PA V2.1, described by APAR PK53163, introduces support for CICS TG statistics (SMF type 111 records). This enables CICS PA users to report on the performance of both CICS TS and CICS TG using the same SMF data source.
CICS TG and Statistic Resource Group Components
CICS TG consists of several interconnected components, with the main one being the Gateway daemon, which listens for network requests from Java and J2EE client applications and routes them onto CICS server regions. Client applications connect to the Gateway daemon using TCP/IP or Secure Sockets Layer (SSL) protocol handlers. These connections are managed by a connection manager component using a pool of threads. Connections from the Gateway daemon to the CICS server regions are allocated from a Worker Thread (WT) resource pool. For more information on the Gateway daemon and recommended architectures and topologies, see the books and whitepapers referenced at the end of this article.
These new statistics provide information about the CICS TG components and protocol handlers. They also provide data on host system environment and connected CICS systems. These statistics are grouped into “resource groups” corresponding to the component for which they provide information. Figure 1shows the CICS TG components and their resource group IDs. Later, we’ll discuss several statistics in more detail in the real-life problem scenario. Details of all the available statistics can be found in the CICS TG information center.
Types of Statistical Data
When CICS TG introduced statistics, it opened the “black box” by providing a window into the operation and status of the Gateway daemon. This insight gives you the capability to perform several activities such as problem determination, capacity planning, monitoring of resource usage, and system tuning. The statistical data provided to help perform these activities consists of current status information about activities in the Gateway daemon, running totals and averages accumulated during the lifetime of the CICS TG, and the start-up configuration values. Accordingly, statistics are categorized into types as described in Figure 2.
Real-Time CICS TG Statistics Function
When an incident occurs, you can now react by analyzing statistical data obtained during the problem period. You can use a z/OS MODIFY command to immediately display CICS TG statistics using the Spool Display and Search Facility (SDSF). The command syntax lets you inquire on individual statistics and on all statistics in specific resource groups. The CICS TG information center contains details on the command syntax. CICS TG V7.1 also lets you filter the information output for statistics of a specific type. Here are some MODIFY commands:
Display all statistics in the Connection Manager (CM) and WT groups:
/F jobname,APPL=STATS,GS=CM:WT
where jobname is the name of the Gateway daemon address space.
Display all interval statistics in the CM and WT groups (Figure 3 shows the output from this command):
/F jobname,APPL=STATS,GS=WT:CM,ST=I
CICS TG Offline Statistics Recording Function
The offline statistics recording complements real-time statistics by recording all statistics values to the SMF at configurable intervals. Interval statistics are reset to their default values at the end of each interval. This makes it possible for you to measure the workload and status of the Gateway daemon in a specific time in the past.
As we’ll see in the real-life scenario, when combined with specialized analysis tools such as CICS PA, SMF recording is an extremely powerful tool for retrospectively diagnosing problems and for tuning system performance based on real workloads.
CICS TG writes statistics to the new SMF record type 111. The statistics values are stored in a collection of fixed length binary data structures. For details on the record format, see the CICS TG information center. You can run a sample application, which ships with the CICS TG, to convert the records into a “human-readable” text format. Alternatively, you can use CICS PA to analyze the records in detail.
You configure the CICS TG to write SMF records by specifying the parameter, STATSRECORDING, in the CICS TG configuration file and by permitting READ access to the BPX.SMF FACILITY class in RACF. You also can configure the frequency that records are cut by using parameters similar to the CICS TS system initialization parameters STATINT and STATEOD. Being able to align CICS TG and CICS TS statistics intervals using these parameters can be useful in configuring similar monitoring policies for comparison purposes.
The following example parameters configure the Gateway daemon to write a record every hour and at 21:00, the end of the business day:
statsrecording=on
statint=010000
stateod=210000
CICS PA Support for CICS TG Statistics
CICS PA provides the following support for analyzing CICS TG statistics (see Figure 4):
• Data reduction: For long-term reporting, you can collect CICS TG statistics (together with CICS TS statistics) in a CICS PA Historical Database (HDB), so you don’t need to keep large SMF data sets.
• Formatted display in ISPF: The CICS PA ISPF dialog presents a formatted display of CICS TG statistics, either from an SMF data set or a CICS PA HDB, with online help describing each statistics field. You can save the formatted displays to a text file.
• Export to spreadsheets or DB2: You can export data from CICS PA HDBs to either Comma-Separated Value (CSV) files, for use with applications such as PC-based spreadsheet software, or DB2 tables, so you can use SQL queries to create custom reports.
Working With CICS TG Statistics in CICS PA
To view a formatted representation of CICS TG statistics in a dumped SMF data set or CICS PA HDB, you specify the data set or HDB name, select a statistics interval (from a list you can filter by a combination of date, APPLID, and interval collection type), and then the CICS TG statistics resource group that you want to view: CICS PA displays the statistics (see Figure 5), with detailed descriptions available for each field.
To collect CICS TG statistics in a CICS PA HDB, you first use the CICS PA ISPF dialog to define an HDB and select the statistics resource groups you want to collect in the HDB. Then you submit a batch job—containing Job Control Language (JCL) that CICS PA generates for you—to load data from a dumped SMF data set into the HDB. You can either use the CICS PA ISPF dialog each time you want to submit a job to load an HDB, or you can use the JCL generated by CICS PA as a starting point for your own jobs.
To export CICS TG statistics to CSV files (for use in PC-based spreadsheet software) or to DB2 tables, you need to specify the intervals that you want to export from an HDB, and the CICS TG statistics resource groups required. Then you would submit a job to perform the export.
For export to DB2, you can use CICS PA to generate JCL for you that, in a single job:
• Loads SMF 111 records from an SMF data set to an HDB
• Calls the DB2 load utility, DSNUTILB, to load those records from the HDB into DB2.
CICS PA also generates JCL for you containing Data Definition Language (DDL) to create the DB2 tables.
Charting CICS TG Statistics in Excel
You can transfer the CSV files generated by CICS PA from z/OS to your PC, and then use them with spreadsheet software such as Microsoft Office Excel to create charts and perform further analysis.
CICS PA Supp or tPa c CP1 2 , “Charting historical CICS performance data” (available from the Web at no charge), contains an Excel add-in that creates interactive charts from CSV files generated by CICS PA (see Figure 6). If your z/OS system allows job submission via File Transfer Protocol (FTP), then, with a single button click, you can use the add-in to:
• Submit a CICS PA batch job to generate a CSV file
• Transfer the CSV file to your PC
• Display a chart of the data in the CSV file.
Analysis of a Real-Life Scenario
Imagine the following scenario: Your Gateway daemons are tuned to achieve an optimum balance between expected Transactions Per Second (TPS) and storage requirements. Usually, performance meets expectations. However, users report slow response times and even time-out errors during periods of peak workload. To make things worse, the peak workloads occur at a time when no one is available to diagnose the problem online using the real-time statistics. Could the offline statistics recording and analysis functionality help you diagnose and resolve the problem?
We set up our test systems to emulate a similar scenario based on the configuration shown in Figure 1. We set statistics recording on with a recording interval of five minutes. The system was tuned to handle a standard workload with periodic peaks in TPS. The CICS transaction we chose to invoke issued a delay call and took 100 ms to complete. We ran an application simulator to drive a standard workload. During our test, we varied the TPS that the simulator drove to rates both below and above the optimal TPS, causing peaks and troughs in the workload. At one point, user response times increased above the expected value and several requests timed out. At the end of the test, we offloaded the SMF records and used CICS PA to investigate the unexpected behavior.
The initial phase of our problem determination approach was to look at an overview of the recorded results to isolate:
• The start and duration of the problem
• The product responsible for the slowdown (the CICS TG, CICS TS or z/OS).
We used the Timeline Chart Excel add-in provided by CICS PA SupportPac CP12 to chart statistical data such as: average response times, time-out counts and peak thread usage indicators, against time. This is an excellent method for gaining an initial overview of the problem. Figure 7 shows the juxtaposition of several charts created by this add-in of key CICS TG metrics. The chart at the top of the figure shows the TPS that the simulator attempted to drive.
The chart of the CICS server average response time (CS_IAVRESP) remained constant at 0.101 seconds for the duration of the test. So the CICS server performed consistently and was not responsible for the uplift in user response time. However, the chart of the Gateway daemon average response time (GD_IAVRESP) varied with the change in TPS. Accordingly, our focus switched to the CICS TG.
Initially, the response time of the Gateway daemon matched the quick response of the CICS server. At 14:09, the Gateway daemon average response rate increased (0.125 seconds) but the increase was within tolerance, as user response times remained acceptable.
At 14:29, the Gateway daemon average response rate rose sharply to nearly 0.9 seconds and remained at this level for approximately 20 minutes. This period coincided with:
• The Gateway daemon timing out transactions waiting for a free WT (WT_ITIMEOUTS)
• A peak in the number of connection managers waiting for WTs (CM_ WAITING).
After 20 minutes, at 14:49, the timeouts stopped occurring and shortly afterward, the GD response time returned to an acceptable level. Interestingly, the chart of the TPS measured shows that it peaks at a little under 100 TPS, but we know the application simulator was attempting to drive around 250 TPS. This illustrates that the system isn’t matching the demand from the client application.
By analyzing the charts in this way, we isolated the time when a problem occurred. Further analysis concentrated on statistics produced by CICS TG components during this timeframe.
We looked at the WT group first because Figure 7 showed an increase in the number of CM threads waiting for WTs at the same time as time-outs and high response times were occurring. Figure 6 shows a CICS PA screen detailing the WT statistics at 14:34. It shows that during this interval:
• The high watermark number of WTs allocated (WT_IALLOCHI) is 10.
• The Gateway timed out 25 times trying to al locate WTs (WT_ ITIMEOUTS).
• The Gateway is configured to allow a maximum of 10 workers (WT_ SMAX).
These findings point to an insufficient number of WTs to process peak workloads. To verify our diagnosis, we increased the number of WTs, restarted the Gateway daemon, and reran our tests. When the workload was increased to 250 TPS, the Gateway daemon average response time (GD_IAVRESP) stayed constant at 101 ms. The problem was resolved. At peak workload, we looked at the WT usage via this MODIFY command:
/F jobname,APPL=STATS,GS=WT
We found that 49 out of our new maximum of 100 WTs were allocated, so we knew we had spare capacity should the workload further increase.
Conclusion
This sample scenario demonstrates that the statistics capability in the latest releases of the CICS TG can provide you with the necessary information to diagnose and resolve performance problems. It also can give you a factual basis to recommend and justify which configuration changes are necessary to resolve the problem, giving a degree of confidence that the changes will work. We also demonstrate that you can resolve a problem, which occurred when nobody was available to monitor the Gateway daemon operation, retrospectively using recorded data.
Long-running systems can produce a large amount of data. We saw that by combining traditional CICS PA functions with the charting SupportPac, you could gain an overview of the available information, quickly isolate which system was responsible for the slowdown, and identify the period of time when it occurred. This speeds up the initial stages of the problem diagnosis and helps you focus your attention on the portion of the data containing the relevant information.
The authors wish to acknowledge Adham Sidhom, Andy Wright, Chris Baker, Colin Westlake, Phil Wakelin, Rob Jones, Richard Davis, and Steve Burghard for their help reviewing the article and providing advice on its technical content.
About the Authors
Simon Knights is an IBM software developer. He has worked extensively on the development of the CICS Transaction Gateway on z/OS.
Email: sknights@uk.ibm.com
Jim Harrison is an IBM software support specialist with more than 27 years supporting and maintaining CICS products. Email: jim_harrison@uk.ibm.com
Graham Hannington works for Fundi Software in Perth, Western Australia. He’s the developer of the CICS PA SupportPac CP12.
