A few tricks from the field…

Jeff A. Bowles
Piccolo Engineering, Inc.

Introduction

This is a "nuts and bolts" talk about several useful tricks for doing build and configuration work using Perforce. Although the tools used are Perforce-centric, there’s no reason that these techniques cannot be discussed or used on other "reasonable" SCM systems.

Version stamping

There’s a long-standing requirement – often unspoken – for software build systems:

This often is taken to mean "compile some nonsense string, such as ‘release 1.2.1.2.1.2.1.2.5’ into the executable," which leaves a beleaguered QA or Tech Support engineer looking through external documents to figure out what files are included in that release.

The objective of this section is to lay out a nifty versioning mechanism that lends itself to Java applications. You should be able to go to a file on your "installation area" or the running installation and run something like this to get the version string:

java -classpath xyz.jar xyz.PrintVersion

Step #1: always have a ‘token’ identifying the source set for a release

There are two reasonable ways to identify the set of source files used to build a release:

Both ways work – even though the first way is easier to explain to a Perforce-naïve manager, the second is more reliable: change numbers cannot be updated after the fact, while labels can. Either way, pick a method and stick to it – the second leverages the Perforce tools a bit more effectively.

Step #2: generate that token before running ‘make’


The build script normally does three things:

The first step would need to make the "token" that describes the "source set" that is being used to build the release, and the build script would pass that to the build tool.

The simplest example is:

  • token=`p4 changes –m1 –s submitted \
    //depot/release1.0/…| cut –d
    ' ' –f2`
  • echo "Building release 1.0 to change #$token"
  • p4 sync //depot/release1.0/…#none
    rm –rf /buildarea/release1.0
    p4 sync //depot/Release1.0/…@$token
  • create that "version string" (this could be a label, also), then the source that corresponds to it.
  • Note that ‘p4 sync #none’ and then ‘rm –rf’ on that tree guarantees that what we build from and what we think we build from are the same thing. (The ‘rm’ kills object files/libraries that aren’t managed by Perforce, to have a clean build.)
  • cd /buildarea/release1.0
    make VERSIONSTRING="release 1.0 $token"
  • Build the product
  • cd /buildarea/release1.0
    make distribution
  • Package it up

 

 

 

Step #3: ‘make’ should use that token to create version information

For java builds, it’s straightforward to include an application inside every .jar file that’s built, which prints out the version string. For example, the file xyz.jar would contain the application xyz.PrintVersion (stored as "xyz.PrintVersion.class") that prints out:

Note how verbose this is allowed to be: if your users are GUI folks who don’t use the CLI, giving the exact commands for an excursion into the command-line realm might be helpful.

This .class file would be built as part of the "make" process, and never included in non-official builds – a red-herring version that says "I’m not an officially-built library!" might be checked in for those cases.

The definitive Java hacker will look at this and say "there are more Java-centric ways to embed version information in the ‘envelope information’ for a .jar file" and would be right. This example is used because it extends a bit more quickly to non-Java environments.

Step #4: tell QA and Tech Support to look for these tokens

If set up this way, then the command:

java -classpath xyz.jar xyz.PrintVersion

becomes the standard way to identify what release you’re running or what’s in an "install area" or CD-ROM ready for installation.

If you’re doing C++ work or C work, embedding the version string in a library is much harder – but it’s still possible to pass this VERSIONSTRING information to the compiler so that "programname –version" prints it out.

The ‘p4 review’ mechanism as a post-submit trigger for ‘autobuilds’

Another build need is to know that what is checked in, compiles. More importantly, the question becomes "does everything else continue to compile with this change included?".

The late Bob Andrews came up with the following approach:

    1. Get the build process down to a few minutes – five or so – first. Incremental builds, for this particular need, might be acceptable.
    2. When a new change (change #YYYY) is submitted to codeline XXXXX, pull it over to a build machine and rebuild everything, looking for errors.

Examples of ‘p4 review’ output

p4 review -t notify:

" Change 1126 jab <jab@jab.steiner> (jab)"

" Change 1127 mike <mike@office>"

p4 reviews -c 1126:

"jab <jab@pobox.com> (Jeff A. Bowles)"

"jojo <jojo@best.com> (Example acct)"

p4 reviews -c 1127:

(nothing)

"p4 review" and "p4 counters" can be used to implement the "state" information that’s needed to do this.

 

A simple review script follows:

Example of flow of a ‘p4 review’ mail-notification script

p4 review -t notify | cut -d' ' -f2 > tmpfile

while read chgnum

do

echo Processing change #$chgnum

reviewers=`p4 reviews -c $chgnum | sed 's/.*<\(.*\)>.*/\1/’ `

p4 describe -s $chgnum | \

Mail -s "change #$chgnum" $reviewers

p4 counter notify $chgnum

done < tmpfile

Example of flow of a ‘p4 review’ autobuild script

p4 review -t autobuild | cut -d' ' -f2 |

while read chgnum

do

echo Building change #$chgnum

p4 sync @$chgnum

make clean # optional

make > make.log 2>&1

if [ $? -eq 0 ] ; then

p4 counter autobuild_knowngood $chgnum

else

Mail -s "build of change #$chgnum failed" admin < make.log

fi

p4 counter autobuild $chgnum

done

Note that the above example rebuilds every change, not just the changes that apply to a particular codeline; it doesn’t cluster the changes together, although that would be fairly straight-forward. It does, however, identify any changes that cause the rest of the system to fail compilation.

Note also that the overnight build doesn’t necessary need to rebuild from the head change number, but the last one that built successfully. "p4 counter autobuild" will tell us which changelist that is.

Do we check in executables or not? Pros and cons.

This is a running discussion on the perforce-user mailing list, and there are arguments in both directions.

There are two sets of requirements that are part of this discussion. Choose one, but not both:

In this religious discussion, it’s important to know what the goals are, and what’s realistic. One of the great truths is that Perforce is an excellent source management system, but is neither a bug database nor a backup/archival system.

If you recreate from source management, but don’t store binaries in repository…

PRO

CON

  • This is a very simple model.
  • There is never a question of whether source in depot matches the corresponding binary.
  • The time to rebuild to create a release or patch might be prohibitive. (This assumes you rebuild the entire release to make a patch – a backup/archival system gets around this.)
  • Developers need to recompile to do basic development.
  • If the tools (compilers) change in the build environment, you have to be aware of those updates/changes prior to making a patch. More things might need to be regressed than you want.
  •  

    Of course, this goes both ways…

    If you store binaries in repository…

    PRO

    CON

  • It’s fast to recreate a build area for building a patch – you aren’t relying on retrieving backup/archival tapes.
  • The developers will love you.
  • "Man who has two watches never knows what time it is."
  • There’s a space cost on the Perforce server – you’re storing much more.
  • The "store one revision only" filetype will deliver a new content to you without your realizing it, when it’s changed – a request for revision #21 will give the same contents as for #21, without warning, if the file is of that ‘filetype’. This can be misleading.
  • So what to do? Most sites pick one model or the other; occasionally, you’ll find a site that stores the final copy of a release (all of it) in Perforce, but no intermediate copies that were handed to QA.