Proposal for SapCat: Site and Platform Customization Administration Tool

R. Lindsay Todd, Ph.D.
P.O. Box 75
Sand Lake, NY 12153-0075
l.todd@acm.org

Abstract

Contents

1  Introduction

The goal of the SC Config competition is a tool to support “platform investigation and program configuration”, a process variously called configuration, customization, or tailoring. One such a tool is GNU autoconf; the SC Config competition is also described as the development of a replacement for autoconf. This document describes our entry to this competition, the Site and Platform Customization Adminstration Tool, SapCat.

We do, of course, entertain hopes of winning this competition. However, we have personally had to deal with (or implement) various autoconf-style tools in various roles, and believe our perspective on this tool's requirments is important. We sincerely hope that the implementers of the ultimate SC Config tool, whether or not it is our entry, will consider our use cases, presented in Chapter 2, and our vision of an installation wizard, presented in Section  3.1.1.1.

1.1  Motivation

We would like to be able to write software that is portable across computer platforms, where a platform is the combination of hardware architecture, operating system, and applications installed on a computer. By portable, we mean that at most minor changes need to be made to the software to successfully compile and install it on any given platform.

But making software portable is difficult, because different platforms vary dramatically. For instance, Unix and Windows are very different. Even the various flavors of Unix have differences, although standards such as Posix have helped considerably to reduce these differences. Programs written in lower-level languages, such as C, must deal with the widely disparate features of different platforms. Higher-level languages, such as Python, Perl, or Tcl, can hide some but not all of the feature differences.

Apart from the feature differences of the computing platforms, systems administrators use diverse site conventions, the local rules specifying where software is installed,^[1] how email is retrieved, which web browser is preferred, etc. Robust portable software must cope with these different site conventions.

These factors conspire to make the development of portable software difficult. But installing portable software can be difficult, too. Often much of the burden of porting the software falls to the person trying installing it on a computer. This person may not necessarily be a programmer ready and willing to spend time on a porting project. This is especially true when software must be ported to a less common platform, or if the computer is administered with different site conventions than the software developer expected.

Sometimes software comes with tools to help port, adapt, or install it. These tools, if not carefully designed, can actually get in the way. Often these tools get any information they need to perform their functions through a GUI. While a GUI may be fine for someone who only occasionally installs software, they tend to be a real nuisance for someone who must install dozens of software systems on several different platforms.

Given the difficulties with porting software, we might ask ourselves, “Is it worth it?” The answer is a qualified “Yes.” By making software portable, it becomes available on more platforms than if it had to be individually developed for each platform. Software that does not interact directly with the operating system can often be easily ported; most of the porting details will have to do with compilers and site conventions. But other software is more difficult to port. The operating system interface supporting graphical user interfaces is vastly different between Windows and Unix; so different, in fact, that any source code relating to the native GUI will have very little in common between these platform. But we hasten to observe that there is very much useful software can be written portably. Furthermore, a system to deal with different site conventions would be useful even when the software is not

Our proposed system, SapCat, deals with the specific problem of customizing software for different platforms. It will generate probes to determine what features are available on different platforms, and make this information available to the software's build procedures. Our system will also provide mechanisms to specify site conventions. For the software developer, our system will make it easier to provide these features than if they had to be developed afresh. For the software installer, our system will provide flexible and simple ways to reliably adapt to the software platform and customize for the site.

Note that there are other pieces that should be provided to enhance our prospects of successfully porting software. One is a common “build” system, such as the result of the SC Build competition. The availability of common programming languages is essential. Libraries providing common APIs to inherently non-portable system interfaces, such as the GUI, can also go a long way to aiding portability. While these area go beyond the scope of the SC Config competition and our SapCat proposal, we nevertheless hope to see further work along these lines.

1.2  Prior art

The idea of creating a tool to assist in adapting software is not new. As mentioned above, there have been other systems (like autoconf) developed to assist in adapting software to different platforms. There have also been many ad hoc schemes devices. Originally, portable Unix software came with a Makefile (or build scripts) and one or more header files that were to be edited by the installer. When this was not enough, more complex systems were devised. In some cases, these systems took on a life of their own (like imake, described below).

We are not familiar with tools developed for the Windows platform. Because Windows provided a widespread and consistent platform, it was far more common to distribute binaries than source code.

In this section, we will survey a few of these systems porting aides.

1.2.1  imake

The imake system comes with the X Window System source distribution. It converts platform-independent Imakefiles into site-specific Makefiles by first determining what sort of system it is running on. Then, central configuration files for that system type are used. These files provide the details needed to build X for a particular platform.

The central configuration files can be adapted for different site conventions, but this is limited by certain assumptions, such as that all X programs are installed in the same location. This assumption may not be bad for X alone, but it becomes a hindrance when imake is used with programs outside the X distribution.

The suggested technique for creating an Imakefile is to find an existing Imakefile that almost does the right thing, and edit that. Using this technique, it is fairly easy to do otherwise difficult tasks, like building shared libraries. But it can be daunting to build an Imakefile from scratch, particularly since it is hard to find all the documentation needed.

The most serious weakness of imake is its requirement that there be configuration files for each platform type. These files are tedious to prepare when new new platforms or new platform dependencies are needed. Consequently, programs end up riddled with tests based on platform names rather than feature availability.

1.2.2  iffe

The iffe system^[2] is a language and programming style to facilitate software portability. Feature test files are run by the iffe script, which interprets feature test files. From these, header files are generated which encapsulate the platform feature information as macros. Source code can include these header file, parameterized with these macros. The iffe script, which with the necessary feature test files usually ships with each package using it, does note appear to affect the Makefiles or otherwise support site conventions.

1.2.3  Larry Wall's metaconfig (dist)

Larry Wall's metaconfig (part of a larger package called dist) creates the Configure script found with Perl. Where iffe had feature test files, metaconfig has units, shell code that tests for platform features. The results of a unit are expressed as shell symbols and C macros. The metaconfig command scans source files for occurrences of these macros and symbols, choosing the units needed to define these (which in turn may depend on other units). The selected units become the bulk of the Configure script.

The resulting Configure script is powerful and highly interactive. It is possible to define the values of symbols on the command line and suppress the interactivity, but it is difficult to determine what symbols are used by a particular Configure script or what those symbols mean. This makes the Configure scripts a nuisance to a systems administrator who does not have the time to sit and interact with the script.

The fact that metaconfig automatically scans source code for symbols is a very nice feature. There are also tools to help wrap shell scripts, header files, and other items that may need symbols substituted into them. We find it surprising that metaconfig has not seen more widespread use.

1.2.4  autoconf

Similar in spirit to metaconfig is the autoconf system, and its companions automake and libtool. autoconf builds a non-interactive Bourne shell script called configure which accepts options to tailor the build system for site conventions, compilation environment, etc. Like iffe and metaconfig Configure, configure compiles and runs probes to discover operating system features. It defines symbols representing features it discovers, and uses these values to transform template header files and Makefiles.

The developer must write a file, configure.in, consisting of GNU m4 macros, to specify the feature probes it needs; autoconf does not scan source code.^[3] The configure.in file is also where the developer can specify searches for other software packages, indicate optional features of the software, and generally perform any other operations neeed to adapt the software.

The related libtool package adds support for shared libraries, otherwise very difficult to do portably. The automake package eases development of the complex Makefile templates used by configure.

autoconf and its companions constitute by far the most successful general customization system. We can identify a number of factors for their sucess:

• autoconf uses tools, like Bourne shell and make, that are already familiar to most Unix developers. (Most autoconf scripts do not actually require GNU m4 programming. Those that do generally need only a little bit, what is documented in the autoconf manual.
• The automake system allows very complicated Makefiles to be constructed with relatively little effort. It really is easier to write the Makefile.am files used by automake than to write reasonably useful Makefiles.
• The generated configure scripts are ideal for busy systems administrators, as they both provide help and do not need continual interactive responses. They are not particularly difficult for the casual user, especially on common platforms where the defaults for options and site conventions are appropriate.
• The autoconf manual is thorough enough to be useful.
• autoconf works well enough to be very useful.
• The GNU project strongly encouraged the use of autoconf.

Nonetheless, there are a few deficiencies with autoconf:

• There is no automatic scanning of source files to determine what feature probes are needed. This is despite the fact that C macros are needed to program in response to probe results. (Last we checked, autoscan would overwrite the configure.in file if used later in project development.) This is an added burden on the programmer.
• Hidden dependencies mean that some probes are executed more than once, through both explicit and implicit invocations. Caching can help, but the result is still a bloated configure script.
• The order of probes in configure.in can be important.
• When additional probes are needed, they require the use of GNU m4 macros with arcane quoting rules.
• It is difficult to debug a probe.

Both imake and autoconf have their proponents, but sane developers have been moving toward autoconf. The need to have produce specialized configuration platforms is too burdensome; it requires the software developer to have access to every platform on which his or her software will be used. But with autoconf, it often happens that a program will build and work the first time it is attempted on a new platform. When there are problems, the solutions usually improve the general portability of the program.

1.3  Our Proposal

Our proposal is a system we call SapCat, which will (a) use project description files to identify source files, parameterized source files, the products of processing source files, installation locations, and any additional options affecting customization of the package, (b) scan the parameterized source files for tailoring symbols, used to tailor for platform features, (c) determine the probes needed to define these tailoring symbols, arranging the probes in correct order, (d) provide an execution environment for conducting those probes, (e) process parameterized files, sustituting the values determined for the customization symbols, and (f) prepare the files needed by build system. In many ways our system resembles metaconfig, with additional features drawn from iffe and autoconf. Section 3 presents detailed description of how SapCat can actually be used.

2  Requirements

In this chapter, we will present the requirements we identified for SapCat.

2.1  Terminology

The process of preparing software to run on a particular platform has been variously called configuring, tailoring, or customizing. Wordsmiths may debate how to best use these terms. Our preference (after studying dictionary definitions) is to call this process of specifying site conventions, specifying local preferences for options, and adapting to the platform, tailoring or customization.^[4] We will call the process of setting up site-wide files, e.g., the sendmail.cf file needed by sendmail, configuration. Finally, we will call the process of setting a user's personal preferences, personalizing.

We call a deliverable unit of software source code a source kit (or frequently, just a kit). A kit is generally distributed in some bundled manner, such as in a Zip, RPM, or compressed tar archive. Such a bundled kits is called a source distribution. A source distribution can be unpacked into a kit, which in turn can be tailored, compiled, and installed on a platform. Alternatively, the installation step can be omitted, and the files that would have been installed instead archived with a tool like RPM, yielding a binary distribution. We could also partially or fully tailor a kit, bundle it up, and produce a tailored source distribution. It may be possible to generate more than one binary or tailored source distribution from a single kit, e.g., to separate the run-time components from development components created from a compiler kit.

The person who is tailoring some software is a tailor (with respect to that software). Since this person will typically also compile and install the software, we may also call him or her the installer. A person who uses software after it has been installed is an end user. Finally, a software developer is a person who creates the software that goes into a kit.

From a software developer's perspective, a kit or source distribution is created from a project. The project consists of all the files in the kit that are not generated by SapCat, as well any other files needed to support development, such as for version control, etc. Some projects may generate more than one kit. Projects may be composed from other projects.

The term package is often used when talking about software entitites. We use this term to refer to a deliverable unit of software in any of its forms, whether that be a kit, a source distribution, or a binary distribution. Often the context will clarify which form of the software is meant.

2.2  The Actors and their Use Cases

We have already introduced a couple actors, the tailor and the software developer. In this section, we will study these actors in more depth. This will help us identify a set of use cases we must satisfy.

2.2.1  Tailors

As we said earlier, a tailor is an actor that customizes a kit. While both tailors and software developers are end users of the SapCat system, we give preference to meeting the needs of tailors, since while we can assume a software developer has some computer expertise, the same cannot be said of the tailor.

Tailors range from the casual installer, who is only interested in acquiring a handful of packages and probably does not want to extensively customize them, to the professional systems administrator, who may need to tailor and install hundreds of packages, perhaps with extensive customization, perhaps for several different operating system platforms, perhaps repeating this process regularly. A third kind of tailor is the systems integrator, responsible for producing binary distributions and tailored source distributions for operating system bundles or other purposes.

2.2.1.1  Casual Installers

The casual installer probably wishes to know as little as possible about the details of tailoring. The self-installing executable wizards prevalent with Windows packages, such as produced by InstallShield, are almost ideal for the casual installer. A GUI is a great help, as are intelligent defaults, completely automated probing of the system, and (when possible) simple suggestions for resolving problems. Binary distributions or tailored source distributions may be preferred by casual installers over pure source distributions.

Perhaps surprisingly to those who are not professional systems administrators, the system administered by casual installers are likely to be the most poorly maintained. Any tool that relied on an accurate database of what other packages have been installed is likely to be inadequate for the casual installer.

The casual installer is also going to be least equipped to create adequate bug reports, whether the failure be in tailoring, compilation, installation, or execution. We need to be able to automatically report as much tailoring information as possible.

The sort of things a casual installer is likely to need to do are:

• Install with Wizard [UC6]
• GUI-controlled Tailoring [UC7]
• Report Tailoring problem [UC23]

2.2.1.2  Professional Systems Administrators

The professional systems administrator is likely to be someone who needs to tailor, build, and install hundreds of packages for many different platforms, sometimes without fully understanding what the package is supposed to do. Often this person is using very specialized site conventions. This person needs to be able to tailor software repeatedly with as little interaction as possible. Being able to script the entire tailoring step is a major convenience. Conversely, a GUI is a major hindrance, unless it can help with scripting. Great flexibility is needed; the professional systems administrator may need to override defaults and probe results. Useful documentation for SapCat is essential.

The professional systems administrator will appreciate a flexible scheme for specifying common site conventions. Since site conventions may be complicated, extension modules (needed at tailoring time) might be needed. Since many packages are being customized and built, it would be useful if there were a way to build and maintain a system-wide cache of SapCat probe results.

If parallel computing facilities^[5] are available, then we will want to be able to exploit this parallelism for both conducting system probes and building the system. We may need to be able to specify what hosts are available, and how remote commands can be started. These conventions may be complicated, and extension modules might be needed to fully exploit parallelism.

The autoconf tool supports a mechanism using the VPATH variable, common to make, to allow one to tailor and build a package in a directory tree other than where the source code is located. This makes it easy to build a package for several different platforms at the same time, a feature very useful to the professional systems administrator. We want to support this feature in SapCat.

A professional systems administrator is likely to be someone who will need to debug any problems encountered during tailoring. A common sort of failure is likely to be a feature probe that gave an erroneous result. (Well, hopefully not too common!) The professional systems administrator will want to simply "repair" the result of that test and re-tailor.

So we can identify a few needs of the professional systems administrator, in addition to those of the casual installer:

• Install a SapCat Prerequisite [UC1]
• Install SapCat [UC2]
• Update SapCat [UC3]
• Update SapCat Components [UC4]
• Update SapCat Probe Cache [UC5]
• Script-controlled Tailoring [UC8]
• Configure for Site Conventions [UC10]
• Configure for Parallelism [UC11]
• Configure Overrides [UC12]
• Install a Package for in Testing [UC13]

2.2.1.3  Systems Integrators

A systems integrator is a tailor who builds binary distributions that can be easily installed by a casual installer. These may go into a binary collection of software, such as one of the many Linux distributions. A systems integrator may also prepare tailored source distributions for an organization.

In addition to the needs of professional systems administrators, a systems integrator has these additional requirements:

• Partial Tailoring [UC9]
• Create a Binary Distribution [UC14]
• Configure Cross-platform Tailoring [UC15]

2.2.2  Software Developers

As we said earlier, a software developer is an actor who creates a source distribution. We assume they are wise enough to use SapCat. There are a number of different kinds of software developers, each with slightly different needs.

One sort of developer is the generic package developer, an actor creating an arbitrary package tailored with SapCat. A special sort of package developer is the SapCat developer.

2.2.2.1  Package Developers

A package developer is generally interested in creating a particular application. This actor may not be expert in the details of software portability or large-scale systems administration; the actor's focus may be on a particular problem domain. But we necessarily require the package developer to accommodate the requirements of SapCat, which must not be too burdensome. We can facilitate compliance by providing mechanisms that will both simplify the software development task and guarantee compliance.

The package developer may discover that platform-specific compiler bugs require special options with certain source files. We need a way to express this sort of information. (The tailor will also need to be able to specify exceptions, and override internal exceptions. Locating these execeptions easily will be important.)

The package developer needs to be able to produce source distributions of all or part of the project. It is important that timestamps on archived files be sane. There may be some changes to source files that should be done when the source distribution is built, like inserting copyrights, licenses, etc.

We need to be able to interface with version control systems. This may require extension modules.

• Make Tailorable Source Files [UC16]
• Create/Modify Package Description [UC17]
• Add a Probe [UC18]
• Test a Probe [UC19]
• Create Kit [UC21]
• Create Source Distribution [UC22]

2.2.2.2  SapCat Developers

A SapCat developer is involved in creating or maintaining SapCat. All the use cases of other software developers apply, as well as these additional cases:

• Update SapCat Probe Documentation
• Test Probe Collection

2.2.3  Use Cases

We will now discuss the use cases we have developed for SapCat. Our design will try to satisfy these use cases. This list tries to be broad, but is undoubtedly incomplete.

2.2.3.1  Maintaining SapCat Software

Several use cases are concerned with installing and maintaining the SapCat software and its configuration on a system. Since SapCat has prerequisites, these also must be considered. Figure 1 shows the relationships between these use cases.

2.2.3.2  Tailoring Source Distributions

The main goal of SapCat is to facilitate software tailoring, so we have use cases relating to the tailoring process. Figure 2 shows the relationships between several of these use cases. A few additional use cases are listed below.

2.2.3.3  Software Development

Figure 3 shows some of the use cases involved with developing software to work with SapCat. These are discussed below.

2.2.3.4  Creating Kits and Source Distributions

A software developer needs to create the source distributions needed by tailors. Figure 4 shows the relationships between these use cases.

2.2.3.5  Problem Reporting

Problems may be encountered as SapCat is developed and new platforms (with new features and bugs) are introduced. SapCat needs a problem reporting mechanism.