Proposal for SapCat: Site and Platform Customization Administration Tool

R. Lindsay Todd, Ph.D.
P.O. Box 75
Sand Lake, NY 12153-0075
l.todd@acm.org

Abstract

Contents

1  Introduction

The goal of the SC Config competition is a tool to support “platform investigation and program configuration”, a process variously called configuration, customization, or tailoring. One such a tool is GNU autoconf; the SC Config competition is also described as the development of a replacement for autoconf. This document describes our entry to this competition, the Site and Platform Customization Adminstration Tool, SapCat.

We do, of course, entertain hopes of winning this competition. However, we have personally had to deal with (or implement) various autoconf-style tools in various roles, and believe our perspective on this tool's requirments is important. We sincerely hope that the implementers of the ultimate SC Config tool, whether or not it is our entry, will consider our use cases, presented in Chapter 2, and our vision of an installation wizard, presented in Section  3.1.1.1.

1.1  Motivation

We would like to be able to write software that is portable across computer platforms, where a platform is the combination of hardware architecture, operating system, and applications installed on a computer. By portable, we mean that at most minor changes need to be made to the software to successfully compile and install it on any given platform.

But making software portable is difficult, because different platforms vary dramatically. For instance, Unix and Windows are very different. Even the various flavors of Unix have differences, although standards such as Posix have helped considerably to reduce these differences. Programs written in lower-level languages, such as C, must deal with the widely disparate features of different platforms. Higher-level languages, such as Python, Perl, or Tcl, can hide some but not all of the feature differences.

Apart from the feature differences of the computing platforms, systems administrators use diverse site conventions, the local rules specifying where software is installed,^[1] how email is retrieved, which web browser is preferred, etc. Robust portable software must cope with these different site conventions.

These factors conspire to make the development of portable software difficult. But installing portable software can be difficult, too. Often much of the burden of porting the software falls to the person trying installing it on a computer. This person may not necessarily be a programmer ready and willing to spend time on a porting project. This is especially true when software must be ported to a less common platform, or if the computer is administered with different site conventions than the software developer expected.

Sometimes software comes with tools to help port, adapt, or install it. These tools, if not carefully designed, can actually get in the way. Often these tools get any information they need to perform their functions through a GUI. While a GUI may be fine for someone who only occasionally installs software, they tend to be a real nuisance for someone who must install dozens of software systems on several different platforms.

Given the difficulties with porting software, we might ask ourselves, “Is it worth it?” The answer is a qualified “Yes.” By making software portable, it becomes available on more platforms than if it had to be individually developed for each platform. Software that does not interact directly with the operating system can often be easily ported; most of the porting details will have to do with compilers and site conventions. But other software is more difficult to port. The operating system interface supporting graphical user interfaces is vastly different between Windows and Unix; so different, in fact, that any source code relating to the native GUI will have very little in common between these platform. But we hasten to observe that there is very much useful software can be written portably. Furthermore, a system to deal with different site conventions would be useful even when the software is not

Our proposed system, SapCat, deals with the specific problem of customizing software for different platforms. It will generate probes to determine what features are available on different platforms, and make this information available to the software's build procedures. Our system will also provide mechanisms to specify site conventions. For the software developer, our system will make it easier to provide these features than if they had to be developed afresh. For the software installer, our system will provide flexible and simple ways to reliably adapt to the software platform and customize for the site.

Note that there are other pieces that should be provided to enhance our prospects of successfully porting software. One is a common “build” system, such as the result of the SC Build competition. The availability of common programming languages is essential. Libraries providing common APIs to inherently non-portable system interfaces, such as the GUI, can also go a long way to aiding portability. While these area go beyond the scope of the SC Config competition and our SapCat proposal, we nevertheless hope to see further work along these lines.

1.2  Prior art

The idea of creating a tool to assist in adapting software is not new. As mentioned above, there have been other systems (like autoconf) developed to assist in adapting software to different platforms. There have also been many ad hoc schemes devices. Originally, portable Unix software came with a Makefile (or build scripts) and one or more header files that were to be edited by the installer. When this was not enough, more complex systems were devised. In some cases, these systems took on a life of their own (like imake, described below).

We are not familiar with tools developed for the Windows platform. Because Windows provided a widespread and consistent platform, it was far more common to distribute binaries than source code.

In this section, we will survey a few of these systems porting aides.

1.2.1  imake

The imake system comes with the X Window System source distribution. It converts platform-independent Imakefiles into site-specific Makefiles by first determining what sort of system it is running on. Then, central configuration files for that system type are used. These files provide the details needed to build X for a particular platform.

The central configuration files can be adapted for different site conventions, but this is limited by certain assumptions, such as that all X programs are installed in the same location. This assumption may not be bad for X alone, but it becomes a hindrance when imake is used with programs outside the X distribution.

The suggested technique for creating an Imakefile is to find an existing Imakefile that almost does the right thing, and edit that. Using this technique, it is fairly easy to do otherwise difficult tasks, like building shared libraries. But it can be daunting to build an Imakefile from scratch, particularly since it is hard to find all the documentation needed.

The most serious weakness of imake is its requirement that there be configuration files for each platform type. These files are tedious to prepare when new new platforms or new platform dependencies are needed. Consequently, programs end up riddled with tests based on platform names rather than feature availability.

1.2.2  iffe

The iffe system^[2] is a language and programming style to facilitate software portability. Feature test files are run by the iffe script, which interprets feature test files. From these, header files are generated which encapsulate the platform feature information as macros. Source code can include these header file, parameterized with these macros. The iffe script, which with the necessary feature test files usually ships with each package using it, does note appear to affect the Makefiles or otherwise support site conventions.

1.2.3  Larry Wall's metaconfig (dist)

Larry Wall's metaconfig (part of a larger package called dist) creates the Configure script found with Perl. Where iffe had feature test files, metaconfig has units, shell code that tests for platform features. The results of a unit are expressed as shell symbols and C macros. The metaconfig command scans source files for occurrences of these macros and symbols, choosing the units needed to define these (which in turn may depend on other units). The selected units become the bulk of the Configure script.

The resulting Configure script is powerful and highly interactive. It is possible to define the values of symbols on the command line and suppress the interactivity, but it is difficult to determine what symbols are used by a particular Configure script or what those symbols mean. This makes the Configure scripts a nuisance to a systems administrator who does not have the time to sit and interact with the script.

The fact that metaconfig automatically scans source code for symbols is a very nice feature. There are also tools to help wrap shell scripts, header files, and other items that may need symbols substituted into them. We find it surprising that metaconfig has not seen more widespread use.

1.2.4  autoconf

Similar in spirit to metaconfig is the autoconf system, and its companions automake and libtool. autoconf builds a non-interactive Bourne shell script called configure which accepts options to tailor the build system for site conventions, compilation environment, etc. Like iffe and metaconfig Configure, configure compiles and runs probes to discover operating system features. It defines symbols representing features it discovers, and uses these values to transform template header files and Makefiles.

The developer must write a file, configure.in, consisting of GNU m4 macros, to specify the feature probes it needs; autoconf does not scan source code.^[3] The configure.in file is also where the developer can specify searches for other software packages, indicate optional features of the software, and generally perform any other operations neeed to adapt the software.

The related libtool package adds support for shared libraries, otherwise very difficult to do portably. The automake package eases development of the complex Makefile templates used by configure.

autoconf and its companions constitute by far the most successful general customization system. We can identify a number of factors for their sucess:

• autoconf uses tools, like Bourne shell and make, that are already familiar to most Unix developers. (Most autoconf scripts do not actually require GNU m4 programming. Those that do generally need only a little bit, what is documented in the autoconf manual.
• The automake system allows very complicated Makefiles to be constructed with relatively little effort. It really is easier to write the Makefile.am files used by automake than to write reasonably useful Makefiles.
• The generated configure scripts are ideal for busy systems administrators, as they both provide help and do not need continual interactive responses. They are not particularly difficult for the casual user, especially on common platforms where the defaults for options and site conventions are appropriate.
• The autoconf manual is thorough enough to be useful.
• autoconf works well enough to be very useful.
• The GNU project strongly encouraged the use of autoconf.

Nonetheless, there are a few deficiencies with autoconf:

• There is no automatic scanning of source files to determine what feature probes are needed. This is despite the fact that C macros are needed to program in response to probe results. (Last we checked, autoscan would overwrite the configure.in file if used later in project development.) This is an added burden on the programmer.
• Hidden dependencies mean that some probes are executed more than once, through both explicit and implicit invocations. Caching can help, but the result is still a bloated configure script.
• The order of probes in configure.in can be important.
• When additional probes are needed, they require the use of GNU m4 macros with arcane quoting rules.
• It is difficult to debug a probe.

Both imake and autoconf have their proponents, but sane developers have been moving toward autoconf. The need to have produce specialized configuration platforms is too burdensome; it requires the software developer to have access to every platform on which his or her software will be used. But with autoconf, it often happens that a program will build and work the first time it is attempted on a new platform. When there are problems, the solutions usually improve the general portability of the program.

1.3  Our Proposal

Our proposal is a system we call SapCat, which will (a) use project description files to identify source files, parameterized source files, the products of processing source files, installation locations, and any additional options affecting customization of the package, (b) scan the parameterized source files for tailoring symbols, used to tailor for platform features, (c) determine the probes needed to define these tailoring symbols, arranging the probes in correct order, (d) provide an execution environment for conducting those probes, (e) process parameterized files, sustituting the values determined for the customization symbols, and (f) prepare the files needed by build system. In many ways our system resembles metaconfig, with additional features drawn from iffe and autoconf. Section 3 presents detailed description of how SapCat can actually be used.

2  Requirements

In this chapter, we will present the requirements we identified for SapCat.

2.1  Terminology

The process of preparing software to run on a particular platform has been variously called configuring, tailoring, or customizing. Wordsmiths may debate how to best use these terms. Our preference (after studying dictionary definitions) is to call this process of specifying site conventions, specifying local preferences for options, and adapting to the platform, tailoring or customization.^[4] We will call the process of setting up site-wide files, e.g., the sendmail.cf file needed by sendmail, configuration. Finally, we will call the process of setting a user's personal preferences, personalizing.

We call a deliverable unit of software source code a source kit (or frequently, just a kit). A kit is generally distributed in some bundled manner, such as in a Zip, RPM, or compressed tar archive. Such a bundled kits is called a source distribution. A source distribution can be unpacked into a kit, which in turn can be tailored, compiled, and installed on a platform. Alternatively, the installation step can be omitted, and the files that would have been installed instead archived with a tool like RPM, yielding a binary distribution. We could also partially or fully tailor a kit, bundle it up, and produce a tailored source distribution. It may be possible to generate more than one binary or tailored source distribution from a single kit, e.g., to separate the run-time components from development components created from a compiler kit.

The person who is tailoring some software is a tailor (with respect to that software). Since this person will typically also compile and install the software, we may also call him or her the installer. A person who uses software after it has been installed is an end user. Finally, a software developer is a person who creates the software that goes into a kit.

From a software developer's perspective, a kit or source distribution is created from a project. The project consists of all the files in the kit that are not generated by SapCat, as well any other files needed to support development, such as for version control, etc. Some projects may generate more than one kit. Projects may be composed from other projects.

The term package is often used when talking about software entitites. We use this term to refer to a deliverable unit of software in any of its forms, whether that be a kit, a source distribution, or a binary distribution. Often the context will clarify which form of the software is meant.

2.2  The Actors and their Use Cases

We have already introduced a couple actors, the tailor and the software developer. In this section, we will study these actors in more depth. This will help us identify a set of use cases we must satisfy.

2.2.1  Tailors

As we said earlier, a tailor is an actor that customizes a kit. While both tailors and software developers are end users of the SapCat system, we give preference to meeting the needs of tailors, since while we can assume a software developer has some computer expertise, the same cannot be said of the tailor.

Tailors range from the casual installer, who is only interested in acquiring a handful of packages and probably does not want to extensively customize them, to the professional systems administrator, who may need to tailor and install hundreds of packages, perhaps with extensive customization, perhaps for several different operating system platforms, perhaps repeating this process regularly. A third kind of tailor is the systems integrator, responsible for producing binary distributions and tailored source distributions for operating system bundles or other purposes.

2.2.1.1  Casual Installers

The casual installer probably wishes to know as little as possible about the details of tailoring. The self-installing executable wizards prevalent with Windows packages, such as produced by InstallShield, are almost ideal for the casual installer. A GUI is a great help, as are intelligent defaults, completely automated probing of the system, and (when possible) simple suggestions for resolving problems. Binary distributions or tailored source distributions may be preferred by casual installers over pure source distributions.

Perhaps surprisingly to those who are not professional systems administrators, the system administered by casual installers are likely to be the most poorly maintained. Any tool that relied on an accurate database of what other packages have been installed is likely to be inadequate for the casual installer.

The casual installer is also going to be least equipped to create adequate bug reports, whether the failure be in tailoring, compilation, installation, or execution. We need to be able to automatically report as much tailoring information as possible.

The sort of things a casual installer is likely to need to do are:

• Install with Wizard [UC6]
• GUI-controlled Tailoring [UC7]
• Report Tailoring problem [UC23]

2.2.1.2  Professional Systems Administrators

The professional systems administrator is likely to be someone who needs to tailor, build, and install hundreds of packages for many different platforms, sometimes without fully understanding what the package is supposed to do. Often this person is using very specialized site conventions. This person needs to be able to tailor software repeatedly with as little interaction as possible. Being able to script the entire tailoring step is a major convenience. Conversely, a GUI is a major hindrance, unless it can help with scripting. Great flexibility is needed; the professional systems administrator may need to override defaults and probe results. Useful documentation for SapCat is essential.

The professional systems administrator will appreciate a flexible scheme for specifying common site conventions. Since site conventions may be complicated, extension modules (needed at tailoring time) might be needed. Since many packages are being customized and built, it would be useful if there were a way to build and maintain a system-wide cache of SapCat probe results.

If parallel computing facilities^[5] are available, then we will want to be able to exploit this parallelism for both conducting system probes and building the system. We may need to be able to specify what hosts are available, and how remote commands can be started. These conventions may be complicated, and extension modules might be needed to fully exploit parallelism.

The autoconf tool supports a mechanism using the VPATH variable, common to make, to allow one to tailor and build a package in a directory tree other than where the source code is located. This makes it easy to build a package for several different platforms at the same time, a feature very useful to the professional systems administrator. We want to support this feature in SapCat.

A professional systems administrator is likely to be someone who will need to debug any problems encountered during tailoring. A common sort of failure is likely to be a feature probe that gave an erroneous result. (Well, hopefully not too common!) The professional systems administrator will want to simply "repair" the result of that test and re-tailor.

So we can identify a few needs of the professional systems administrator, in addition to those of the casual installer:

• Install a SapCat Prerequisite [UC1]
• Install SapCat [UC2]
• Update SapCat [UC3]
• Update SapCat Components [UC4]
• Update SapCat Probe Cache [UC5]
• Script-controlled Tailoring [UC8]
• Configure for Site Conventions [UC10]
• Configure for Parallelism [UC11]
• Configure Overrides [UC12]
• Install a Package for in Testing [UC13]

2.2.1.3  Systems Integrators

A systems integrator is a tailor who builds binary distributions that can be easily installed by a casual installer. These may go into a binary collection of software, such as one of the many Linux distributions. A systems integrator may also prepare tailored source distributions for an organization.

In addition to the needs of professional systems administrators, a systems integrator has these additional requirements:

• Partial Tailoring [UC9]
• Create a Binary Distribution [UC14]
• Configure Cross-platform Tailoring [UC15]

2.2.2  Software Developers

As we said earlier, a software developer is an actor who creates a source distribution. We assume they are wise enough to use SapCat. There are a number of different kinds of software developers, each with slightly different needs.

One sort of developer is the generic package developer, an actor creating an arbitrary package tailored with SapCat. A special sort of package developer is the SapCat developer.

2.2.2.1  Package Developers

A package developer is generally interested in creating a particular application. This actor may not be expert in the details of software portability or large-scale systems administration; the actor's focus may be on a particular problem domain. But we necessarily require the package developer to accommodate the requirements of SapCat, which must not be too burdensome. We can facilitate compliance by providing mechanisms that will both simplify the software development task and guarantee compliance.

The package developer may discover that platform-specific compiler bugs require special options with certain source files. We need a way to express this sort of information. (The tailor will also need to be able to specify exceptions, and override internal exceptions. Locating these execeptions easily will be important.)

The package developer needs to be able to produce source distributions of all or part of the project. It is important that timestamps on archived files be sane. There may be some changes to source files that should be done when the source distribution is built, like inserting copyrights, licenses, etc.

We need to be able to interface with version control systems. This may require extension modules.

• Make Tailorable Source Files [UC16]
• Create/Modify Package Description [UC17]
• Add a Probe [UC18]
• Test a Probe [UC19]
• Create Kit [UC21]
• Create Source Distribution [UC22]

2.2.2.2  SapCat Developers

A SapCat developer is involved in creating or maintaining SapCat. All the use cases of other software developers apply, as well as these additional cases:

• Update SapCat Probe Documentation
• Test Probe Collection

2.2.3  Use Cases

We will now discuss the use cases we have developed for SapCat. Our design will try to satisfy these use cases. This list tries to be broad, but is undoubtedly incomplete.

2.2.3.1  Maintaining SapCat Software

Several use cases are concerned with installing and maintaining the SapCat software and its configuration on a system. Since SapCat has prerequisites, these also must be considered. Figure 1 shows the relationships between these use cases.

2.2.3.2  Tailoring Source Distributions

The main goal of SapCat is to facilitate software tailoring, so we have use cases relating to the tailoring process. Figure 2 shows the relationships between several of these use cases. A few additional use cases are listed below.

2.2.3.3  Software Development

Figure 3 shows some of the use cases involved with developing software to work with SapCat. These are discussed below.

2.2.3.4  Creating Kits and Source Distributions

A software developer needs to create the source distributions needed by tailors. Figure 4 shows the relationships between these use cases.

2.2.3.5  Problem Reporting

Problems may be encountered as SapCat is developed and new platforms (with new features and bugs) are introduced. SapCat needs a problem reporting mechanism.

3  Overview of SapCat

This chapter is introduces the SapCat tool. Whereas Chapter 2 introduced the requirements for SapCat, this chapter discusses scenarios of how SapCat could actually be used to meet these requirments.

SapCat comes in both GUI and command line variants. The GUI command can be invoked with no arguments, in which case it opens a window with menus leading to all its various operations, or it may be invoked (perhaps through a Windows association) with a Package Description Database file given as a parameter, in which case it begins a dialog for tailoring the kit. The command line version also specify the name of a Package Description Database file, followed by switches controlling tailoring features, to indicate that a kit should be tailored. It may also be specified with other switches to invoke other operations. Everything that can be done through the GUI can also be done through the command line version.

A tailor has quite a different perception of the SapCat tool than a software developer. A tailor is interested in using SapCat to tailor software, usually to compile and install it, possible to create binary distributions. A software developer is mainly concerned about creating source distributions, getting everything correct so that SapCat will function for the tailor. Of course, a software developer will probably tailor projects in the course of development and testing.

Because these two perspectives are very different, we will present overviews from each of these points of view. We will consider the view of the tailor first, since SapCat is primarily intended to serve the tailor.

3.1  Tailors' Perspective of SapCat

As we discussed in section  2.2.1, tailors range from casual installers to professional systems administrators to systems integrators. Each of these has different ways of using SapCat

3.1.1  Casual Installers Use of SapCat

We expect casual installer to run SapCat on systems where it has already been installed, either as part of the operating system, by a professional systems administrator, or possibly through installation of a binary distribution. In the sections that follow, it should be understood that SapCat may have some system-wide configuration files that alter the default behaviour (generally to support site conventions).

3.1.1.1  Installation through the SapCat Wizard

Ideally, a casual installer can receive a source distribution and simply apply start SapCat in its “wizard” mode. With Windows, this would be done by receiving a Zip file and unpacking it; then clicking on the SapCat icon in the kit (which would be associated with the SapCat project file for the package).^[7] A casual installer on a Linux might be able to operate in the same manner. People working on legacy Unix systems might need to start SapCat from a command line.

Once the SapCat GUI started, it presents a pane with some information about the package, like its name, version, and a brief description. Buttons appear to indicate whether to “continue” or “cancel” installation. There is also a button for entering “advanced” mode.

Since we are talking about the casual installer, we assume that the “continue” button is chosen, and the pane shows the next view. If there is information that must be entered by everyone, the next few panes will present opportunities to enter this data. Each pane will present buttons to “continue”, “cancel”, get “help”, or go “back”.

After all needed data are entered, the view changes to show the status of the process of probing the operating system. Should this step fail, a problem report would be presented, along with any information that may have been supplied with the package to help in this case, and an opportunity to send a bug report to the package's maintainers.

Once probing has completed, the status view changes to show the progress of actually tailoring the kit. Then a pane would be presented giving the tailor the chance to “build” the package or to “quit”.

We assume here that the “build” button is chosen. The package is now built, and a pane shows the progress. As with the probing phase, we present a problem report if this step fails. (It is almost certain that a problem in this step should be reported to the package maintainer.)

If the package includes any self-tests, a pane appears allowing testing to proceed or be skipped. After testing, another pane appears allowing a choice of installing or quitting.

Installation is an interesting problem on multiuser systems, since there are likely access controls preventing a system-wide installation. Furthermore, the software is likely to need to be tailored differently for each of these cases. Probably the choice of installation type will have been presented as one of the questions asked when data was collected.

After installation, SapCat will offer to clean the kit directory of unnecessary files (such as object code).

The package may have tests to check if installation was successful. If so, the tailor will be given a chance to run these tests. (Note that if the kit directory is not first cleaned, there is a possibility that these tests will falsely succeed due to the presence of some file that should have been removed!)

Our hope is that it will be as easy to install Open Source software using SapCat as it is to install binary distributions with wizards on Windows today. But we also hope eventually to do even better than we have described here, and support getting package updates, automatically getting prerequisites, etc.

3.1.1.2  Using the SapCat GUI

Even a casual installer may decide to change some of the default decisions made by the SapCat wizard. This can be done by clicking on the “advanced” button on the initial pane.

Once this is done, a pane appears with “tabs” to select different pages. The initial page allows entry of data needed to tailor the package, installation directory locations, prerequisite package locations, and option choices (if any) for the package. A second page lists each of the probes used in tailoring this package, allowing the tailor to override their results.

After these choices are made, the “continue” button can be pressed, and probing begins. The tailor can choose to build, install, and test the package.

3.1.2  Professional Systems Administrators Use of SapCat

A professional systems administrator may use SapCat just like a casual installer. However, a professional systems administrator may wish to use the command line interface, so that SapCat can be used in scripts. A professional systems administrator may also need to install SapCat on systems where it is not already available, configure site conventions, and configure for parallelism.

3.1.2.1  Using the SapCat Command Line Interface

Both the command line and GUI versions of SapCat accept various “help” options that can be used to determine what customization options are available in a kit, what probes will be executed, and what site conventions are tested. The GUI version can even be used to create a file containing the selected options and probe result overrides that will automatically be used by SapCat when it is invoked. Alternatively, once the professional systems administrator has determined the available options, he or she can construct an appropriate command line invocation.

The command line interface is not as “wizardly” as the GUI. It performs a distinct task, such as getting help text, tailoring (probing and adapting source files), invoking the build system, installing, etc. But this level of control is what is needed for effective scripting.

3.1.2.2  Bootstrapping SapCat

Usecases UC1 and UC2 detail the need for a professional systems administrator to be able to install SapCat or a prerequisite on a system that does not already have SapCat. The various packages need to be installed in the correct order. Each package would have one or more bootstrap scripts; it might be necessary to invoke the appropriate one for the platform (e.g., a batch file on Windows or a Bourne shell script on Unix).

Other packages could ship with bootstrap scripts besides those needed to get SapCat running. However, these may introduce requirments that do not exist when they are not used. For instance, a package implemented in Java is not likely to need a C compiler. However, the bootstrap script may need a C compiler to build tools needed to perform its tasks.

3.1.2.3  Maintaining SapCat

There are a number ways in which SapCat needs periodic maintenance. We would expect that a casual installer would use RPM packages (or similar) to keep SapCat maintained on a system. We hope to provide tools to make it easier to maintain SapCat from source.

One thing that all tailors are likely to need to do periodically is update the cache of probe results. This obviously needs to be done if we update the probe collection. However, operating system updates can also invalidate the probe cache. So can adding or updating other packages, such as libraries.^[8] SapCat has an option for updating this cache.

3.1.2.4  Site Conventions and other Customizations

When a tailor runs SapCat, it is possible to specify every directory in which the package will be installed. Like autoconf, many of these directory paths will take on default values relative to $prefix and $exec_prefix, which in turn will take on platform-specific default values. But if desired, extension modules, written in Python, can be added to compute alternate default values, parameterized on the package name, version number, architecture, etc. Other common parameters can be given site-specific default values.

3.1.3  Systems Integrators Use of SapCat

A systems integrator prepares binary distributions for other users. This probably, goes beyond the scope of the actual SC Config competition, but it makes sense to incorporate some support within SapCat for a couple of reasons.

First, the installed components of some packages may divide nicely into several different binary distributions. For instance, a kit for the X Window System might create clients, development tools, X servers, and runtime libraries.^[9] The software developer is in the position of best knowing how a package could be divided into distributions, but it is the systems integrator that actually creates distributions. It makes sense to include this information in the Package Description Database.

Second, determining what distribution builders are available on a platform is a tailoring operation, as is customizing the build system to be able to build distributions. It is also possible to extend knowledge of other build systems to SapCat.

A systems integrator would tailor and build a kit as appropriate. However, instead of allowing SapCat to install the package, he or she would instead invoke SapCat with its distribution builder option. From the command line interface, this would need to request which distribution builder is to be used, and which binary distributions should be constructed. From the GUI, menus and checkboxes would be used to identify how and what to construct.

As we mentioned above, a professional systems administrator may create a file containing tailoring information to be used when SapCat later tailors the kit. A systems integrator may create this file and create a tailored source distribution, in much the same way as a software developer creates a source distribution.

3.2  Software Developers' Perspective of SapCat

A software developer creates source distributions. Our focus will be on how to incorporate SapCat.

SapCat relies on a Package Description Database to specify what is built, which source files are needed, what options may be specified, etc. From this information, SapCat determines what files need to be scanned for symbols defined by probes. This is used to determine what probes will be used at tailoring time. This database can be distributed over multiple directories, since a Package Description Database file can specify that other directories are included.

Source files must be parameterized based on macros or other symbols that SapCat can identify as indicating the results of a feature test or package option specification. Complete information on the probes included with SapCat will be available in the documentation.

Ultimately, it may be useful to provide one or more libraries of common operations that programmers can use. These libraries would provide uniform interfaces, while themselves being parameterized on operating system features. These libraries would also simplify operations like building path names relative to options specified when the package was tailored. However, portable librares are not a pancea for writing portable software, since they usually incur a performance hit.

4  Design

In this chapter, we present a design for the SapCat system. We start with a high-level overview of the different packages that constitute SapCat. Then we will go into more detail about the interfaces between packages, how the packages operate, etc.

4.1  Components

The SapCat system is composed of several components, as shown in Figure 5. The functions of the main components are:

SapCat GUI

The SapCat GUI module provides the GUI for the SapCat system. As Figure 5 shows, it functions by interacting with the SapCat Manager module.

Package Description Database

The Package Description Database module provides the interface from various component to the actual package database. We show this as a single module, but expect that alternative implementations will arise. For instance, the actual database can be stored in XML, a Python-like language, or LDAP.

SapCat Manager

The SapCat Manager component is the main component. It interacts with the various databases, probes the operating system, modifies files, etc.

Tailoring Database

The SapCat Manager accesses the Tailoring Database to find cached results, site conventions, etc. Results of operating system probes and options specified by the tailor are stored in the Tailoring Database. This module provides an interface to the actual database, which is distributed over several locations.

Build System

The Build System module is an interface to the actual build systems. It may be invoked by the SapCat Manager, but it may also run independently. We show it depending on the Package Description Database and the Tailoring Database; from these two databases, the build dependencies can be determined. Figure 6 shows how actual build systems are related to the generic Build System module.

Figure 6:  Package Diagram for Build System

Probe Database

The Probe Database contains information about the various feature probes that may be needed.

Distribution Builder

The Distribution Builder is used to create source distributions, tailored source distributions, and binary distributions. It uses the Build System to make sure that everything to go into the distribution is first built. It uses data from the Package Description Database and the Tailoring Database to create the actual distributions. It may be invoked by the SapCat Manager. Figure 7 shows how actual distribution builders are related the the generic Distribution Builder module.

Figure 7:  Package Diagram for Distribution Builder

Bootstrap Script Builder

The Bootstrap Script Builder creates bootstrap scripts. This is a generic module; Figure  8 shows actual bootstrap script builders relate.

Figure 8:  Package Diagram for Bootstrap Script Builder

File Scanner

The File Scanner scans source files for symbols defined by probes. For C and other such languages, template header files are created. In other cases, File Scanner reads existing template files.

File Adapter

The File Adapter module transforms template files that need to be adapted for the platform. How this is done may depend on language dependencies.

When a bootstrap script is used, the File Adapter may not be available. A simple adapter, written in C, will be available that works with a few languages. Bootstrap scripts cannot be created for packages that cannot use this simple adapter.

Language Module

Figure 9:  The Language Module

The Language Module provides language-independent routines for the File Scanner and File Adapter. This abstract module uses the actual language-dependent modules. Figure 9 shows how these modules relate.

How do these modules actually interoperate? The following sections describes a number of operations.

4.1.1  Tailoring

The Package Description Database and the Probe Database form the heart of SapCat. From the Package Description Database, SapCat determines the source files. The File Scanner is used to find what symbols or macros are defined are defined in these files. From this list of symbols, the SapCat Manager module can assemble a list of probes and the order in which they must execute.

The SapCat Manager examines command line switches and options files, then begins evaluating probes. When possible, probe results come from the probe results cache, through the Tailoring Database; otherwise the probe is executed, and the result stored in the Tailoring Database.

When probing is completed, the Package Description Database is scanned for files that need to be adapted, and the File Adapter module actually creates the modified files. Then the Build System module is used to create configurations files for the actual build system. At this point, tailoring is complete.

4.1.2  Building Software

SapCat is not directly responsible for building the software in a kit. However, it provides a uniform interface to build systems. This may involve creating Makefiles at tailoring time. The SapCat Manager will use the Build System module to configure and run the build system.

4.1.3  Bootstrap Scripts

As with tailoring, the SapCat Manager determines what probes are needed. But this time, the Bootstrap Script Builder module is given the list of probes. Inquiries to the Package Description Database are made, and a script is constructed to carry out these probes. Inquiries are also made to the Package Description Database module to add commands to the script to build and use a simple adapter, written in C, to adapt files when the script is run.

When a bootstrap script is run, it checks for command line switches or an option file for tailoring information. Then it executes the probes. Finally, it runs the adapter.

We have not yet decided how the build system will interoperate with bootstrap scripts. We will probably have options to configure either Makefiles or scripts.

4.1.4  Creating Source Distributions and Binary Distributions

From the Package Description Database, SapCat Manager can determine what needs to go into a source distribution. SapCat is configured with information about what local distribution builders (tar, zip, etc.) are available. The SapCat Manager calls upon the Distribution Builder module to actually create the distribution.

For a binary distribution, the SapCat Manager first calls upon the Build System module to make sure everything needed is up-to-date. Then it uses the Package Description Database to determine what goes into the binary distribution. Finally, the SapCat Manager calls upon the Distribution Builder module to create the binary distribution.

Some kits may contain the components of multiple distributions. The Package Description Database contains everything needed to determine what files go into which distributions. It also contains information about any prerequisites packages.

The Distribution Builder module can be extended using Python modules to support additional distribution builders.

4.2  Interfaces

This section describes the interfaces to the various modules comprising SapCat. Sadly, it is not complete.

4.2.1  Package Description Database

Figure 10 shows our current thoughts for the Package Description Database. It consists of representations of file system objects, both source and constructed. It also contains some information regarding package options and distribution tags. Not yet modelled is information for additional probes and platform-specific information.

We have not settled on a final syntax for the Package Description Database. In some ways, this is less important than the object interfaces to this module. Indeed, there is merit to having several possible syntaxes. A colleague who promotes LDAP observes that LDAP would allow the database to be distributed over multiple sites (and probably provide fast access).

4.2.2  Probe Database

Figure 11 shows the Package Description Database. Probes can be selected on the basis of symbol names, operating system class, and programming language. (Operating system class is a very general category, such as Unix, Windows, etc.) The probe contains a test program to test for some feature.

The difficulty of designing the probe database is that we cannot assume any particular scripting language is available to help us to interpret probe results. This difficulty arises from the need to support bootstrap scripts. We might be able to divide probes into two categories: those that can be used by bootstrap scripts, and those that cannot (but may use Python scripting).

We have not chosen an actual syntax for probes. We expect each probe to be a file residing in a directory on a search path; perhaps each probe file will be described in XML. We should point out that our LDAP advocate believes that LDAP would form a better basis for the probe database, and this could lead to a distributed system wherein the latest available versions of probes are always used.

4.2.3  Tailoring Database

Figure 12 shows the Tailoring Database. Before a probe is conducted, the Tailoring Database is consulted. Likewise, the File Adapter uses the Tailoring Database to extract symbol values.

In reality, the Tailoring Database is implemented over several databases. There is a system-wide cache, which should be protected so that only trusted users can regenerate it. When a kit is tailored, probe information is cached in the kit directory. Finally, a tailor may manually enter information into a configuration file in a kit to provide values to SapCat.

4.2.4  Remaining Modules

We have not completed our design of the remaining modules. Most of them are fairly simple. The most complex module is the SapCat Manager. However, the only module that depends on it is the SapCat GUI, and that dependency will be through the command line interface.

5  Development Plan

As specified by the SC Config guidelines, SapCat will be implemented mainly in Python. There are a few components, mainly relating to the bootstrap problem, that may be implemented in C.

The design presented in Chapter  4 divided SapCat into a number of different modules. Work could be divided amongst several developers along these boundaries. Indeed, SapCat is a large enough system that several developers will be needed. Should we be unsuccessful in getting additional developers, our plan is to offer a Python course and recruit students to help develop it.

5.1  Phase 1

Our initial focus will be implementing the “essential” use cases. A realization of the Package Description Database must first be chosen, since this component is so central to the rest of SapCat. The abstract generic interface, and an XML implementation, can probably be constructed in a couple of days (including a rudimentary test framework).

The File Scanner, File Adapter and Language Module modules are pretty simple. Language support for simple template files (substituting @SYM@, like autoconf) can be designed pretty quickly. Similarly, a prototypical Probe Database can be constructed quickly.

With the addition of a simple versions of the SapCat Manager, Build System, and Tailoring Database, we should have a working prototype of SapCat. With a single programmer working part-time, it should take a month or three to complete this phase.

5.2  Phase 2

After a working prototype has been developed, it needs to be “polished” into production code. The Tailoring Database needs to include caching; the File Scanner and File Adapter modules need to be enhanced to work with C header files, and the Build System needs to developed to work with Makefiles. During this time, the {:scPD} system will also be expanded.

5.3  Phase 3

After the command line version of SapCat is working well, it is time to add the GUI version. Initially, this will invoke individual SapCat operations from the command line version. But once this work, the wizard dialog needs to be developed.

After the GUI is functional, remaining features can be implemented.

5.4  Testing Methodology

There are several ways to approach testing with SapCat. First, each module will be equipped with internal consistency checks that can be invoked as desired by the SapCat developer. Secondly, there will be regression tests designed for the modules.

The most difficult aspect of testing is the probe database. The testing of the module code itself can be arranged through a regression test, but the actual probes give results that necessarily vary by platform. However, we should be able to design a way to execute every available probe on a platform, recording the results. These results can be regression-tested against that platform. With the help of a distributed team of SapCat testers, we should be able to get pretty wide coverage of platforms. If nothing else, being able to view all probe results can be helpful in determining whether or not a probe actually works.

Finally, software developers may need to add new probes. SapCat will provide a way for individual probes (and their prerequisites) to be run, so that the developer can verify that it is working correctly.

6  Summary and Acknowledgements

We have presented a proposal and design for a software tailoring tool called SapCat. This tool helps programmers write portable code, by providing feature tests and macros they may use in their programs, scanning their programs to determine which features need be tested, assisting in writing portable build scripts, and creating source distributions. SapCat helps busy system administrators by providing the mechanism needed to tailor and build packages requiring a great control, without needing constant interactions. It helps system administrators unfamiliar with a package to determine what features need customizization, thus assisting them in making intelligent choices. It helps systems integrators create binary distributions. Finally, our design of an installation wizard should make it installing Open Source software nearly as simple, and far more flexible, than it currently is to install binary packages on Windows or Linux.

On a personal note, we have found the Software Carpentry competition extremely stimulating and rewarding. We hope to see more further competitions of this sort, and look forward to seeing the fruit of this first round.

I would also like to thank my colleagues at Rensselaer Polytechnic Institute with whom I have had many enjoyable discussions about software customization issues, particularly Mike Douglas, Garance Drosehn, Alan Powell, and Mike Sofka. I also want to thank my manager, Gary Schwartz, for allowing me to take “vacation” time during a particularly busy time, and my family, who have graciously allowed me many hours to hide in the “abyss” to work on this proposal.

Footnotes

[1]	For example, on many Unix systems, programs are installed into the /usr/local/bin directory, or some other common location. But some sites store every program in its own directory. See, for instance, how Rensselaer Polytechnic Institute stores applications individually in its AFS cell in a structure called /campus.
[2]	Glenn S. Fowler, David G. Korn, John J. Snyder, Kiem-Phong Vo, Feature-Based Portability, In Proceedings of the USENIX Symposium on Very High Level Languages, pp. 197-207, October, 1994.
[3]	The autoconf tool does include a script called autoscan, but currently this usually does only part of the job.
[4]	We recognize that “configuration” would be a fine term for this process. But our choice of terminology now will affect our natural choices of file names, etc. later. We thought it desirable to ultimately use different names than autoconf and metaconfig had used.
[5]	We are using the term loosely here to describe both true multiprocessing parallel systems and loosely-coupled clusters of very similar systems.
[6]	If there is a good way to run scripts on native Windows, I would love to hear about it. Neither I nor my associates are familiar with any such features.
[7]	It might be possible to have some sort of archive that unpacks and automatically starts SapCat. This would be better yet.
[8]	Why not rely on packages to also provide updates to the probe cache? Because this information is sensitive to the gestalt of all packages installed on the running system, which may not be the same as the system on which a binary distribution was built.
[9]	It always irks me that IBM's AIX insists on putting X servers on systems with no graphics systems, since that is part of the “X11 run-time environment”. Apparently someone forgot that X is networked.

---

R. Lindsay Todd

Last modified: Fri Jun 30, 2000