While working on my next column, the prospect of reviewing the SLES 11 release developed into a significantly larger project than I had anticipated. Therefore, instead of my usual “Linux on System z” column in this issue, I present you with this article, which details the many features of SUSE Linux Enterprise Server 11 (SLES 11).

Novell launched SLES 11 on March 24, 2009. As is typical of such version updates, close cooperation with IBM and others resulted in numerous specific enhancements for IBM hardware and software. IBM requested approximately 100 features specific to System z. This article examines many of those changes as well as features of general interest on all hardware platforms.

Software Packaging

One of the more frequent complaints about SUSE Linux Enterprise is the large number and size of packages required because of inter-package dependencies or Novell’s choice of compile options. For SLES 11 a number of individual packages were broken into several packages, as is commonly done with other distributions such as Debian and Fedora. The intent was to provide core functionality in one package, which would require few or no other packages. Other packages might provide extra functions—a graphical interface, for example—which would then require a large number of other packages to be installed. This approach enables system administrators to have more control over how “bloated” their systems become.

A second approach taken was to create an extremely minimal system pattern, mainly intended for appliances. This configuration pattern installs roughly 260 packages, taking up less than 600MB of disk space. It Initial Program Loads (IPLs) with very little additional function; however, it forms a much more attractive basis for creating a stripped-down standard image.

In SLES 11, the kernel has been split into multiple Red Hat Package Manager (RPM) packages:

• Kernel-default-base contains the actual kernel and a small number of modules for things such as Small Computer System Interface (SCSI) and the EXT3 file system.

• Kernel-default includes all the other kernel modules and particularly the hardware-dependent ones.

• Kernel-default-man contains actual man pages explaining the various messages emitted from the System z-specific kernel modules. This message catalog could be considered a very rough beginning of a Linux for System z messages and codes manual.

One significant change that enterprise customers should be aware of is that the packages relating to High Availability (HA) clustering and Mono have been split into separate “extensions.” System z subscribers will automatically have access to these extensions. Subscribers from other platforms, however, will have to pay extra for them, unless they meet the conditions to be “grandfathered” in at no additional cost. A future article will cover this in more detail.

The Software Development Kit (SDK) has always been an unsupported collection of software. Over time, many unrelated packages were added that also weren’t supported, but were thought to be useful to customers. However, the SDK is no longer a “dumping ground,” as such; it has been slimmed down to include only those packages that are in line with the purpose of an SDK. It will remain unsupported as before.

The service tooling has picked up a new “extras” software channel. This channel has been added to contain software that users might find useful, but lacks Novell support. There’s not much in that channel at the moment, but it’s expected to expand. The software update “stack” has been modified to allow multiple, concurrently installed versions of a package. This is primarily intended for kernels, but there are likely to be more uses.

The command line software update tool (zypper) can now operate on RPMs from a pre-defined repository, an arbitrary Uniform Resource Identifier (URI), or a file on the local system, all with automatic dependency resolution. 

Here’s a quick look at some of the more noticeable changes in the package mix:

Added/updated:

• openAIS and Pacemaker replaced Heartbeat2 for HA clustering. A few pieces that were retained are useful regardless of the overall clustering technology used. The packages used for HA were moved to the new HA extension product (SLE HA 11).

• IBM Java 1.4.2 was updated to java-1_ 4_2-ibm-1.4.2_sr12.

• IBM java-1_6_0-ibm-1.6.0

• ruby-1.8.7: Note this isn’t Ruby on Rails, just the Ruby development language itself. Ruby on Rails and other so-called “Web 2.0” development tools are included with the SDK, which means they’re unsupported.

• File System in User Space (FUSE): While FUSE has been available since SLES 10, SLES 11 ships with several file systems that exploit it. Some examples are:

- curlftpfs: mount FTP servers

- encfs: layered file encryption

- fuseiso: mount iso, img, bin, mdf, and nrg CD-ROM images

- fusesmb: mount, with full browser support, a network neighborhood

- sshfs: mount over ssh

- wdfs: mount of WebDAV shares.

• “Command not found” handling in the shell. This is a double-edged sword. For the experienced system administrator, getting a semi-verbose message about every typo he or she makes can be irritating. For the new system administrator, typing in a command (correctly), and being told which package he or she needs to install to be able to execute that command can be helpful. The experienced system administrator will likely quickly discover how to disable it.

• Supportutils: For some time, Novell Technical Services has asked system administrators to install this package to gather information for debugging purposes. SLES 11 includes supportutils on the installation media, so updates to it are received through the normal update channel.

Removed:

• The Mono application framework was moved to the new Mono Extension product, SLE Mono. The parts of Mono needed for normal system use were retained in SLES itself. The full Mono project RPMs for all platforms continue to be available directly from the Mono Project Website without charge.

• IBM java-1_5_0-ibm-1.5.0

• SPident: This command was dropped for several reasons, including that it often was inaccurate or misleading. The replacement for it, Supportability Analysis Module (suse-sam), isn’t intended for direct use by system administrators; rather, it’s a tool that other Novell packages, such as the supportutils previously mentioned, invoke. Accordingly, the only method to determine whether a system is completely up-to-date that’s really left to the system administrator is to look at the contents of /etc/SuSE-release.

• GNU Privacy Guard (GPG) Version 1

• Previously marked for removal:

- CPINT (in favor of VMCP from s390 tools)

- Journaled File System (JFS) utils

- Enterprise Volume Management System (EVMS)

- rug/zmd

- uw-imapd.

z/VM Integration

The eXecute In Place file system (xip2fs) and Discontiguous Saved Segment (DCSS) block device drivers were updated to work with DCSSs with addresses above 2GB. This will let applications have one or more shared objects whose cumulative size is greater than 2GB. In combination with the elimination of Linux struct pages, this allows DCSSs to become freely usable anywhere in the kernel address space without excessive control block overhead. This gives z/VM systems programmers the freedom to create DCSSs at the high end of z/VM’s addressable virtual storage instead of trying to identify which z/VM guest that will use the DCSS has the largest virtual storage size to allocate the DCSS just above that.

Before SLES 11, typical z/VM systems programmer actions such as detaching and reattaching a device to a guest operating system could result in Linux treating a device as not accessible when it really was accessible. Improved handling of dynamic subchannel mapping will help eliminate this problem.

A frequently requested feature—the ability to enter extra kernel parameters when IPLing Linux on z/VM—is now available via the z/VM PARM keyword on the IPL command:

IPL devno PARM 1

IPL devno PARM init=/bin/bash

This feature is very convenient when:

• Booting the system into single-user mode

• Specifying init=/bin/bash to work around some start-up problem

• Entering arbitrary kernel parameters that are needed only for this IPL, such as cio_ignore.

Installer

Announced at the same time as the z10, the Open Systems Adapter (OSA) Express 3 has four ports, two for each Channel Path Identifier (CHPID). When SLES 10 SP2 was released, the kernel was able to use those two additional ports, but neither Yet another Setup Tool (YaST) nor the installer could. With SLES 11, both have been updated to allow use of these ports. The installer also accepts a new parmfile entry of Portno=0 or Portno=1. If this isn’t provided in the parmfile, the installer will prompt for it.

For those of you with strict security requirements, support for encryption of the root file system during installation is now available (just be careful to remember the passphrase that lets you mount it during IPL). Note that this doesn’t mean file contents will be encrypted while the file system is mounted. The application owner or system administrator is still required to perform file-level encryption, or to implement full-filesystem encryption using other tools postinstallation. (An encrypted file system makes the entire file system unavailable to anyone who might obtain physical access to the server and remove the media containing it.)

When SLES 10 SP2 was released, YaST and the installer were for all platforms— changed so persistent device by-id names were used in /etc/fstab. Although this change was of little note for most architectures, it was a significant mistake for Linux on System z, as it burned physical disk ids into specific images, making it impossible to relocate minidisks to other volumes without significant effort or clone from a “gold master” image. SLES 11 updated the default /etc/fstab entries to use “bypath” for System z, which shouldn’t interfere with the various published cloning methods or relocating minidisks using the standard VM tools.

Installation onto multi-pathed devices is now possible. Previously, deactivation of all but one of the paths to the Storage Area Network (SAN) was required.

The procedure for IPLing the installation kernel, parmfile, and initrd on z/VM has been documented for a long time. Some people, however, aren’t comfortable with Xedit, so a technique borrowed from the SLES 10 Starter System included a CMS REXX EXEC on the installation media in the /boot/s390x/ directory named sles11.exec that assists in the initial start-up process. Like the three other files needed to start an installation, it must be uploaded to the z/VM system.

Network Configuration

The installer and YaST were modified to allow different host names for different IP addresses on different Network Interface Controllers (NICs). Also, there’s now support for channel bonding qeth devices in YaST. Channel bonding is essentially taking two separate network interfaces and “bonding” them together to act as one interface to provide redundancy or additional bandwidth. Setting up the bonding has worked in the past, but occurred manually.

One change that will annoy those who like to get down into the details of system configuration files is that hardware configuration information no longer resides in the /etc/sysconfig/ hardware/ directory. Instead, everything now occurs in udev rules in /etc/ udev/rules.d. Due to the arcane and complex nature of udev rules, if you lack confidence in your ability to control udev, you should use YaST to configure all hardware devices, including network interfaces.

System Management/Configuration

Many, if not all, of the kernel modules IBM maintains will now emit messages with message number prefixes. In addition, the source code was scanned to create man pages for all these messages. It isn’t the same as the typical IBM messages and codes manuals, but it’s a step toward improving the documentation system administrators need. IBM also created a PDF document with all the messages and explanations. It’s available from www.ibm.com/developerworks/linux/linux390/documentation_ dev.html by clicking on the Kernel Messages - SC34-2599-00 link.

SLES 11 supports dynamic memory attachment and detachment. For systems with stringent uptime requirements, this allows more flexibility in managing assigned resources. For systems running in a Logical Partition (LPAR), Linux also can use the “assign storage” and “attach storage element” Service Call Logical Processor (SCLP) commands to attach memory to the image. This supports use of standby memory introduced with the z10.

Similarly, the kernel in SLES 11 can use the “configure CPU” and “deconfigure CPU” SCLP commands to switch CPUs between the standby and operating state when they’re enabled or disabled with the CPU online attribute. This lets Linux use standby CPUs that aren’t visible until they’re activated.

The multi-pathing support in device-mapper was updated to enable a new version of IBM’s GDPS/PPRC Multiplatform Resiliency disaster recovery offering. This new version will support site failover and Hyperswap (transparent storage subsystem failover) to Linux running in a System z LPAR and non-mainframe Linux images attached to an IBM enterprise storage array. This also provides support for replicated SCSI/FCP disks to be failed over.

There are times when it’s necessary to cancel an I/O before the usual hardware timeout limits occur. This might be in an HA cluster or a disaster recovery event. Previously, there was no way to force this under the Linux kernel. With SLES 11, that’s now possible. IBM’s xDR and device mapper-based RAID1 with real-time enhancements are two possible uses for this capability. Others are expected to emerge later.

The ability of mainframes to “call home” when a problem arises has long been one of its strengths compared to other architectures and manufacturers. SLES 11 allows a site to decide if it wants to add some information to this data and, to a certain extent, what that information is. Four new pseudo files have been added to the sysfs pseudo file system:

• /sys/firmware/cpi/sysplex_name

• /sys/firmware/cpi/system_level

• /sys/firmware/cpi/system_name

• /sys/firmware/cpi/system_type.

The contents of these files are placed there at each IPL by the /etc/rc.d/boot. cpi init script. The 8-byte string values for sysplex_name and system_name come from /etc/sysconfig/cpi, which the system administrator can modify. The value of system_level is a hexadecimal number composed of the currently running kernel version. For example, 2.6.27 would be translated to 02 06 1b. Currently, system_type is set to LINUX. If this information isn’t desired, it can be disabled for the next system IPL by editing /etc/sysconfig/cpi. This is really useful only for systems running in an LPAR. Any call home data this feature provides will be visible from the Hardware Management Console (HMC).

You can now dynamically add or remove cryptographic cards to an LPAR or z/VM guest. This will allow crypto capacity to be adjusted as needed without interrupting running applications.

You can now re-IPL from a device other than the one you initially IPLed. This is helpful during installation, but also can be useful in normal operations. The new IPL device is specified and queried via the lsreipl and chreipl commands in the s390-tools package. The sysfs files involved are under the /sys/ firmware/reipl/ hierarchy.

The Linux stand-alone dumper can now write the dump to SCSI devices. This is useful to SCSI-only shops and shops running with large virtual storage systems that would exceed DASD capacity.

Prior to SLES 11, the Linux Kernel Crash Dump (LKCD) tool processed Linux dumps. The LKCD is being replaced with a newer tool called lcrash, which was enhanced to support the Linux for System z stand-alone dumper and provide support for cross-architecture debugging. So, no matter where the dump analyst is most comfortable working, lcrash will support debugging of dumps from other architectures.

Kernel Virtual Machine (KVM) is available as a technology preview in SLES 11. Technology previews, included for Novell customer convenience, aren’t supported, and at this point probably aren’t suitable for production use; they offer early adopters a chance to “play” with the new technologies and provide feedback to help improve possible future releases. Currently, there are no tools for System z to exploit KVM and budget cuts at IBM Germany make the future of KVM exploitation on System z uncertain. There is an experimental, unsupported proof-of-concept program named “kuli” on IBM’s developerWorks site at www.ibm.com/developerworks/linux/linux390/kuli.html, but comments on the page make it clear that kuli isn’t for the faint of heart.

This article has discussed only a portion of what’s new. A follow-up article will examine file systems, hardware support, Fibre Channel Protocol improvements, performance management, security, and two new extensions to SUSE Linux Enterprise Server.