Laura Wingerd
Sybase, Inc.
wingerd@sybase.com

Christopher Seiwald
Perforce Software, Inc.
seiwald@perforce.com

March , 1997

Abstract

1. Introduction

In this paper we describe how jam -- a software construction tool -- builds the large, commercial Sybase database and connectivity products. We discuss the hurdles that had to be overcome in bringing jam to Sybase and, no less in difficulty, in bringing Sybase to jam.

Sybase Corporation produces the Sybase Open Server and associated connectivity products. Little about the size or scope of the Sybase system is trivial. Surpassing 10,000 source files and 3 million lines of code, Sybase is among the larger software systems in the world (though by no means the largest). Its size is compounded by its complexity: evolving over the last 10 years and having been ported to just about every major (and sometimes not-so-major) platform, the Sybase code is the product of hundreds of separate engineering efforts.

Like many other companies faced with the problem of building large software products, Sybase built a home-brew tool around the familiar Make program. The result, called Sybmake, was a halo of UNIX shell scripts which would dynamically combine product- and platform-specific Makefile fragments and then run Make on the result. And like many other such build systems, Sybmake was a monster. Because it ran shell scripts to construct Makefiles, it was slow -- compiling a single file involved 180 UNIX fork() system calls. Sybmake performed header-file analysis (using a variant of GNUs makedepend utility), but as it could only process one directory at a time, it would often reprocess header files that were used commonly throughout the system. The build information for compiling even a single file often came from a dozen different sources -- Makefile fragments, shell scripts, special variable files, as well as the user's environment -- rendering the process largely incomprehensible. And being a shell script concoction, it was difficult to port, especially to non-UNIX platforms, of which Sybase had many.

Faced with the rising complexity, decreasing maintainability, and dropping speed of Sybmake, Sybase looked for a replacement tool that would serve as a foundation for a "Universal Make." Because we had experience with jam, we proposed using it.

Jam is a freely available Make-like program. Originally written by one of us (Seiwald) while at Ingres Corporation, jam is an attempt to render the building of complex systems manageable while leaving simple systems simple. The essence of jam is a rule language for describing the often meandering relationships between source files and their built targets. Unlike Make, whose simple language can only build the dependency graph one piece at a time, jam supports user-defined relationships. The rules associated with each type of relationship set up the actual dependency graph. For example, in this jam statement:

The Main rule states that the source.c and source2.c are to be compiled and linked into program. Main itself uses other rules (the Link and Cc rules, among others) to arrange the details. Along the way, these user-defined rules make use of built-in directives to create the dependency graph.

The definitions of user-defined relationships, such as Main, are collected into a ruleset file called the Jambase. Once there, they can be used and reused for any source files and build targets sharing the same type of relationship. Typically the user-defined rules are called from a Jamfile that sits in the same directory as the source files. With this arrangement, the Jambase provides a build system interface, and the Jamfile is a client of this interface. (Jam comes with a stock Jambase that can at the very least build jam itself.)

Jam's language contains features that enable it to understand source code hierarchies whole. Rather than recursively invoking itself on subdirectories, jam reads all Jamfiles directly, building the dependency graph for all source files and build targets in the system. Jam uses the dependency information to avoid building targets if targets on which they depend failed to build, rather than proceeding blindly ahead or halting altogether. With full dependency information, jam can also parallelize the execution of whole builds.

Jam's language is rich enough to do without a preprocessor: it has flow-of-control statements, variables, and many other features of general purpose languages, although it falls short of a general purpose langauge itself.

Jam was well poised to eliminate most of Symakes shortcomings, and a full-scale conversion to jam was undertaken.

Jam was invented to solve problems such as Sybase's, but its applicability could only be verified by collecting requirements, analyzing them, and then judging the results. The requirements themselves broke down into two categories: functional requirement and user requirements.

The functional requirements that jam had to meet were on the surface simple: jam had to be able to reproduce the results of the existing Sybmake system, while making modest gains in speed and simplicity. Unfortunately, as is often the case with large products, no single person knew how the Sybase products were built, and virtually no functional documentation was available. This meant that most of the functional requirements had to be reverse engineered by observing Sybmake in action.

Collecting the user requirements makes a much different,though equally familiar, story. Invariably, engineers will complain that their build tools need drastic improvement while in the same breath asserting that nothing should change. What they want is the same tools, somehow working better. To avoid this connundrum, we had to engage in a certain amount of "requirements combatting," in effect trying to set users' expectations to reflect jam's approach.

As we shall now describe, we were able to meet our basic goal of building Sybase with jam, by both customizing jam for Sybase and reworking Sybase to fit jam. The result is a case study in both the use of jam as a build tool and in replacing the build tool underneath a large software product.

2. Problems and Solutions

Building a product with jam is a matter of writing a set of jam rule definitions that describe the ways in which the pieces fit together, and then writing rule invocations that reference the actual source files and build targets. The bulk of the effort goes to the rule definition.

Jam comes with a set of rule definitions as part of the stock Jambase file. The Jambase rules work very well for developing prototypes and for work on new software projects, but they don't handle the historic complexities found in legacy build systems.

Using Jambase as a starting point, we developed a set of Sybase-specific rules to solve several challenging build problems.

A crucial problem we faced was that of referencing source files with pathnames.

First, the root of a codeline is not constant, but instead reflects the local workspace. Our Sybase-specific rules would have to refer to files using some kind of relocatable root so that users could build from a codeline copy in any location.

Jambase provides a rule call SubDir which computes pathnames based on an environment variable which points to the top of a codeline. The SubDir rule uses a platform-independent list of directory levels and constructs absolute pathnames in the native platform syntax. These pathnames are used to set the jam special variables LOCATE and SEARCH.

However, in large codelines, the absolute path name to reference a source file is likely to be long. On NT, long path names in compile and link commands can exceed the command line length limit and require use of cumbersome "response files" to issue commands.

Finally, Sybase codelines use a consistent and particular layout to allow platform-specific source files to overshadow generic source files of the same name. This structure requires searching for every source file in two places, a platform-specific location and a generic source location, and prevented us from making use of the SubDir rule, in any case.

Our solution was the creation of the SSPart ("Sybase-specific part") rule, which is like the SubDir rule in that it sets LOCATE- and SEARCH-related variables. But instead of constructing rooted paths, it constructs the relative paths of source files, as computed from the current directory in effect for each jam invocation.

A compile action that would have been constructed thus, using fully rooted pathnames:

could now be constructed using pathnames relative to the directory in which the jam command was invoked. For example, if the user's current directory were /work1/main/svr/sql/generic, the above action would be constructed thus:

We've already said that Sybase builds a large number of large software products on a large number of platforms. Compiler flags, link libraries, include paths, etc., must be configured for each platform. A given Sybase product is likely to exist in several concurrent codeline branches, each of which requires specific release level information to be configured. And each of those product/platform/branch combinations is also built in as many as six "build modes," which include an optimized mode, a symbolic debugging mode, a code coverage test mode, etc.

The typical if-then-else treatment of configuration variables becomes unmanageable with this many build combinations.

With jam, we were able to consolidate all of this into what was essentially a flat file containing a configuration table. After extending jam's argument list interface (see "Feedback to Jam", below), we implemented the "SSConfig" rule to construct the configuration table.

For example, the following fragment of a [hypothetical] configuration table defines some of the differences between a Solaris build and an NT build. The fields, delimited by ":", are platform list, build mode list, and product list:

The configuration table is stored in the codeline, along with source code and jam makefiles.

Not surprisingly, the Sybase codelines contained several instances of like-named source files (e.g., "source/utils/foo.c" and "source/svr/foo.c"). Make-based build tools aren't bothered by this kind of duplication, because they deal with only a single directory at a time. But jam uses filenames as unique identifiers of targets across the whole source tree.

To accommodate this, the Jambase rules qualify the filenames they encounter with grist (a uniqueness prefix) constructed from arguments of the last SubDir rule invoked. We had problems adapting this to the Sybase codelines because there were also several situations in which a single file needed to be referenced by rules in more than one directory context, or built in more than one way by rules in a single directory context.

The solution we arrived at was to abandon any implicit grist computations (i.e., we pitched the SOURCE_GRIST variable in Jambase) and allow users to put their own explicit grist on filenames in their Jamfiles. We added logic to our Sybase-specific rules that would detect and report duplicate targets, but we let the users decide what uniqueness prefixes to use and where.

For example, Jamfiles that contained:

and

could be modified by the developer to read:

and

Developers had no trouble understanding the purpose of these uniqueness prefixes (once we stopped calling them "grist") and jam had no trouble using the explicit grist to differentiate the targets.

Like many complex products, Sybase is built from some source files which are generated by tools or processes invoked during the build. In some cases these files are generated by standard tools (e.g., yacc), but in some cases the files were created by one-of-a-kind processes.

We developed several source generator rules (e.g., SSYacc, SSLex, SSConcat, SSAwk, SSSed, and the very general SSFilter), but some source files still required special actions or rules. While we would have liked to come up with a complete set of rules to build all Sybase products, we did not want to spend too much time trying to normalize the few historically evolved, one-of-a-kind processes. As a compromise, we provided an "SSGenerateSource" rule which would appear to have a standard interface in the jam makefile which invoked it, e.g.,

but in fact was a lookup table indexed on the target. Inside the SSGenerateSource rule was a switch statement to do the one-of-a-kind processing needed for each target, e.g.:

At the outset of the jam build system rollout we maintained control of the SSGenerateSource rule source code, but eventually dropped tailored versions of it into each product codeline, to be maintained by developers responsible for the files it generated. We hope developers will see that it's much easier to use a standard rule to generate a source file than to code one-of-a-kind entries in the SSGenerateSource rule, and that the SSGenerateSource will eventually become obsolete.

The legacy build system at Sybase required some "pre-build" steps to be run on "blessed" platforms known to provide stable and consistent versions of yacc and lex. As one would expect, this was a productivity bottleneck, because developers had to wait for nightly builds to complete on the blessed machines in order to get updated yacc and lex outputs to use on the platforms they were working on.

Our goal was to produce a build system that would allow developers to build any Sybase product consistently and reliably, on any plaform, strictly from source. Unfortunately for us, yacc and lex outputs vary from platform to platform, and indeed, some platforms on which Sybase is built don't even have yacc or lex.

The "blessed" machine for yacc had been Solaris, because the Solaris yacc supports a special flag to replace the generic parser template with a Sybase-maintained custom one. We modified the Berkeley yacc to provide this functionality, and built it on all platforms where Sybase is built.

The "blessed" lex machine was SunOS 4.x, but only because a built version of Gnu's flex happened to be available there. There was also some post-processing which needed to be done to the flex output in the Sybase build to accommodate Sybase-specific source code requirements. So we customized flex to make the post-processing unecessary, and built this customized flex on all Sybase platforms.

By building uniform, customized yacc and lex programs on all platforms, we eliminated the uncertainties presented by native yacc and lex, and obviated the need to pre-build on designated platforms.

Several very large archive libraries are generated in building Sybase products. Some of these libraries are so large (60Mb) that simply re-archiving an single object in a symbolic debug-mode build can take ten or fifteen minutes on certain platforms. We took two approaches to handling archive libraries that ultimately shortened the build time and added versatility to the build system.

First, it was clear that builds would benefit from a reduction in the size of the largest archive library, jam or no jam. We split this library into a seventeen smaller libraries, simply dividing it up along source code directory lines. (This was a one-time operation, where we replaced library target names in jam makefiles. We wrote a program which analyzed symbol reference dependencies and calculated a minimal link order for the seventeen libraries that had once been a single library. Our final link command for one executable contained multiple references to the seventeen libraries, for a total of twenty-six references in all. Suprisingly, referencing libraries multiple times did not noticeably slow linking time, while not having any one large library sped up build time considerably.)

Second, we replaced Jambase's Library rule with set of rules to build "just-in-time" libraries. When archive libraries are large, it's faster to delete them and recreate them from existing compiled objects than it is to re-archive updated compiled objects in existing archives. In our just-in-time library paradigm, all compiled objects are saved in directories corresponding to the libraries into which they will be archived. At link time, the archives are created. Subsequently, any compiled object that is updated causes the affected library to be deleted, and any subsequent link dependent on that library causes the library to be recreated.

Not only did this shorten rebuild time in the "fix a single source file, recompile, and relink" scenario, but it gave us a set of compiled objects we could reuse in the case where we needed to link a shared object library from the same source files.

Sybase codelines are large and complex, and our Sybase-specific rules are succinct. Naturally, Sybase developers are going to make a few mistakes, wrong assumptions, and typos when updating their Jamfiles. Jam's control language, efficient and powerful as it is, doesn't provide complete syntax or error checking, and there is no rule in Jambase that forcibly exits the parsing phase. A mis-typed Jamfile can produce thousands of "Warning: no targets on rule..." messages in a large build system.

We added two facilities to mitigate any frustration that might result from jam's unrestrictive grammar. First, we implemented an internal "SSErrorExit" rule. All of our Sybase-specific rules check arguments for correctness and call SSErrorExit if necessary. In addition to displaying any message passed to it by the caller (e.g., "SSLibrary rule was invoked with no source file list") SSErrorExit uses variables set by the last SSPart rule encountered to display the relative path to the Jamfile likely to be the offender.

Second, we implemented a set of trace flags users can set, primarily to to audit results of the configuration table, but also to display duplicate filenames, warn about obsolete or ambiguous configurations, etc. Users set the trace flags on the jam command line, e.g.:

This produced output that, at least, aids the jam maintainers in debugging user problems.

3. Feedback to Jam - Jam 2.2

Building Sybase was a learning experience for jam. Since we had the liberty of changing jam itself to suit our needs, we took the opportunity whenever the result seemed generally useful. We plan to roll these changes into the next release of jam, 2.2.

Originally, jam's custom rule language was intended solely for building the dependency graph, relating target files and their sources. It is, however, also possible to write utility rules whose only effects are the setting of variables and the calling of other rules. This works because a rule's targets and sources are only treated as actual files if updating actions get associated with them. Rules without associated actions can be viewed as subroutines written in the jam language.

As the number of utility rules grew in the Sybase effort, a limitation of the language became apparent: only two parameters -- the targets and sources -- could be passed to a rule. Some utility rules needed more parameters -- perhaps two sets of arguments and the name of an output variable to be set.

To satisfy this, the strict "Rule targets : sources ;" syntax of jam's rule invocations was relaxed to allow multiple :'s. Basically the :'s became a generic parameter separator. Within a rule, the positional parameters are referred to with $(1), $(2), $(3), etc, with $(1) and $(2) being synonyms for $(<) and $(>) (targets and sources, respectively).

The biggest user of this extended syntax is the SSConfig rule, described above, which uses 5 parameters to decide how to set configuration specific variables.

Jam has a fairly powerful target binding algorithm where, during the course of Jamfile parsing, targets can be referred to without a relative path, only to have the actual path determined when actions are executed. For example, "cc -o $(<) $(>)" would expand the target $(<) and source $(>) to the real object and source files. Unfortunately, this feature was only applied to the targets and sources of a rule. It was difficult to apply the binding operation to other variables, because it wasn't possible to tell if a variable referred to a file or was just arbitrary text (such as compiler flags).

This limitation was most pressing when linking executables:

Ideally, $(<), $(>), and $(NEEDLIBS) would all be bound to real targets before being expanded, because they all referred to buildable targets. The other variables should just be treated as text. But only $(<) and $(>) were bound; $(NEEDLIBS) had to be the literal pathnames of the libraries to link.

The simple solution adopted was to add a "bind" modifier to the actions, so that the indicated variables should be bound:

This turned out to be useful in many situtations where ancilliary sources, such as build tools and macro files, needed to be referred to in the actions. By allowing these variables to be bound as targets, we could leverage jam's ability to find tools scattered throughout the build tree, rather than relying on absolute paths.

The jam rollout at Sybase is initially half a dozen platforms; another dozen or so are soon to follow. Distributing the jam componentry to these platforms is in itself a challenge. A large part of the problem is distributing the Sybase-specific Jambase files: distributing executables is easy enough, but to distribute Jambase files requires either that we always put the files in the same place or rely on an environment variable to point to them. Neither approach is appealing. Systems administration at Sybase is such that ensuring that /usr/local/lib exists on all systems is nearly impossible, and systems such as VMS and NT have no "standard" location for such support files. Environment variables we've sought to avoid all along: currently jam can run without any special environment.

The approach we took simplified this problem dramatically: we compiled the Jambase text into the jam executable itself. This built-in Jambase behaves just as an external Jambase file would and can be overridden on the command line (should users want to try their own). The result is that instead of having to deliver an executable and a handful of auxilliary files, we provide a standalone executable (called "sjam") for each platform.

4. Unsolved problems

Sybase is a Clearcase shop, and although the jam build system we put together is completely independent of the source management system, there were two problems Clearcase posed that we did not have good solutions for.

First, when a developer works in a Clearcase view, it is possible for for him or her to change the contents of a source file in such a way that jam can't detect the change. This happens when the Clearcase "uncheckout" command is used and a file is reverted to an earlier state, which, unfortunately for jam, includes an earlier timestamp. If the developer had compiled an object file from the source file before the uncheckout, jam would have no way of knowing that the object file now needs to be updated, because the object file is still newer than its source file.

One way to handle this might be by impementing a Clearcase trigger to write a marker file that jam could detect. E.g., the command "ct uncheckout foo.c" would write a view-private zero-length "foo.unco" file, and our rules would specify that foo.o depends on foo.unco. Supporting this rule for header files would be little more complex. Also, some kind of scrubber would required to remove marker files that are no longer needed due to subsequent checkins from other views.

A second problem presented by Clearcase was the time it takes to read files stored in a VOB. One of the most powerful features of jam is its ability to scan for "include" directives in source files. In a medium-sized Sybase codeline, the same jam scan that takes 45 seconds in a local codeline copy and 90 seconds on an NFS-mounted codeline copy can take from 2 to 4 minutes in a Clearcase view.

NT is used as a development platform for some Sybase products, and is a release platform for most Sybase products. In a software company the size of Sybase, NT tools and environments become regionalized, where, for example, one product group relies on one version of the Microsoft C compiler, another group relies on another version of it, while a third group relies on the Watcom compiler.

It's difficult to provide a set of jam actions to accommodate all the NT tool variations. Our configuration table helped with minor differences between versions, but at the time of this writing our build system on NT was usable only with Microsoft compilers and linkers. A good long run solution would be to provide a set of action templates that can be customized for any vendor's compiler and utilities.

Several jam limitations came to light as we implemented our build system on VMS Alpha. Most of these limitations had to do with variable modifiers and VMS filename handling, and have now been fixed in jam. Our remaining problem is that VMS command status codes don't seem to map cleanly to "success" or "failure", causing some of our build actions to appear to fail when they've actually succeeded.

There is a bug in jam wherein rules that update more than one target with a single action invocation could cause builds to get out of sync when using parallel processes (e.g., "jam -j4"). Our workaround for this was to add a rule to touch the codependent files individually, which prevented builds from failing but did add a little bit of undesirable overhead.

5. Conclusions

The rollout of jam at Sybase is currently underway, and we can make some observations and draw some conclusions.

The first important observation is that jam is on track to achieve its primary goal at Sybase: portable builds across all platforms. Currently several flavors of UNIX and NT are built with jam. VMS, OS/2, and further UNIX ports are underway. Users on all platforms see the same tool with the same behavior, using the same (compiled-in) Jambase and the same Jamfiles.

One of the deficiencies with Sybase's previous build tool, Sybmake, was that it alone was unable to comprehend wholly the system to be built. It instead required either the user or a script to step around directory by directory in order to get the pieces built in the right order.

In constrast, jam is capable of comprehending systems in total. A single invocation of jam can read all the Jamfiles in the system, construct a single large dependency graph, and then proceed to execute the actions to build the system.

Jam's complete comprehension of a build is best exemplified when it is invoked the second time, just after a build is complete. For Sybase's connectivity codeline, jam's output consists only of the message "...found 6208 target(s)...", signifying two things: that it found every source and target file in the build, and that everything is up-to-date. The output of any Sybmake run, in contrast, was often in the thousands of lines, which made it difficult for engineers to determine what, if anything, was happening.

A goal of using jam was to improve the performance, as well as the functionality, of the build system. In one case -- full builds from scratch -- Sybmake and jam were on a par. This is because the run time is dominated by compiling and linking, rather than any overhead introduced by the build tool.

Build of Sql Server Products

In all other cases, jam was either slightly or significantly faster than Sybmake. Jam had two great advantages. First, while Sybmake was an amalgamation of shell scripts, executable programs, and the UNIX tool make(1), jam is a single monolithic executable. With jam, only one executable needs to be run, instead of hundreds. Second, because jam builds the system whole, it only processes each source file once. Sybmake, on the other hand, stepped from directory to directory, re-reading the contents of the directories and searching for header file dependencies repeatedly on the same sets of files.

Possibly the most telling number is the last in the table above: it represents what a developer must normally endure after making a change before he or she can try it out. The psychological difference between waiting two minutes for jam or 10 minutes for Sybmake is more than just a factor of 5: developers can wait through a two minute delay without losing focus. When forced to wait 10 minutes, they often "find something else to do" to fill their time.

Where jam really pulled ahead was in its ability to do parallel builds, updating multiple independent targets at the same time. Because jam had complete dependency information, it is able to achieve complete parallelization. Sybmake could take advantage of GNU make's ability to parallelize updates, but only within a single directory. Further, this feature was limited to UNIX. Except for a limitation caused by a jam bug (as noted above), jam could completely parallelize builds on both UNIX and NT. The parallelized build on a Solaris machine with two processors was just about half (60 minutes) of the single process build. We're expecting more impressive numbers from the 4 processor SGI machine.

The time it takes to convert to a new build system isn't to be slighted. The conversion at Sybase began in January of 1996 and rollout was underway as of November. The largest factor in conversion was the discovery of how the existing system worked, followed by the coding of the replacement implementation for jam. Much of the coding time was actually testing time: to test builds, unfortunately, requires building the same products over and over again, often from scratch. With a build time ranging from just over an hour (for the server) to several hours (for the connectivity products), easily 3/4ths of the coding time was spent waiting for test builds to complete.

Users planning on replacing a build system should expect another obstacle to completion: moving targets. During the course of the jam implementation Sybase went through several product cycles and some rescheduling of priorities. As a result, the target of jam's initial rollout changed twice.

The most taxing problem, psychologically at least, was trying to meet developers' expectations of how the build system should work. Largely their requests and requirements were reasonable, but nonetheless difficult to work into the initial rollout. Balancing developers' wishes with a schedule that could be met took its toll in the form of extra meetings and miscellaneous communication, but in the end it aided acceptance of jam by its new customers.

Jam is now being used by several groups within Sybase to build the Sybase server and connectivity products, with more groups scheduled to join shortly.

After the initial retraining for the developers, we expect jam to pay off well: developers will be able to use the same tool on all platforms, allowing Sybase to move its currently UNIX-heavy environment more towards NT; jam's ability to reproduce builds (with minimal output) means less time will be spent tracking down phantom build bugs; and the raw speed of jam for full and partial builds will save developers' precious time and patience.

The payoff for jam is equally promising: many generic improvements, some described above, are being incorporated into the next release (jam 2.2). We expect these new features to be readily reusable the next time a company decides to re-engineer its build system.

6. Bibliography

Atria Software, "Building Software Systems with ClearMake", ClearCase Users Manual, Natick MA, May 1994.

Geoffrey M. Clemm, The Odin Reference Manual, available via anonymous FTP from ftp.cs.colorado.edu.

S. I. Feldman, "Make - A Program for Maintaining Computer Programs", BSD NET2 documentation, April 1986 (revision).

Glenn Fowler, "The Fourth Generation Make", Proceedings of the USENIX Summer Conference, June 1985.

Peter Miller, "Cook - A File Construction Tool", Volume 26, comp.sources.unix archives, 1993.

Christopher Seiwald, "Jam -- Make(1) Redux", Usenix UNIX Applications Development Symposium, Toronto, Canada April 1994.

Richard M. Stallman and Roland McGrath, "GNU Make - A Program for Directed Recompilation", Free Software Foundation, 1991.

Zoltan Somogyi, "Cake, a Fifth Generation Version of Make", Australian Unix System User Group Newsletter, April 1987.

Dennis Vadura, dmake(1) manual page, Volume 27, comp.sources.misc archives, 1990.