Peter Jackson, software architect, Symbian Ltd.
Introducing a new SCM system in a growing company is an opportunity to redesign the development process. That is what Symbian did in the summer of 1999. This paper describes how Perforce was chosen and the profound effect it had on Symbian's development processes. It is intended to help organisations which may be contemplating similar changes or who just want to compare and contrast their development processes with others.
Symbian produces an operating system for Wireless Information Devices such as Communicators and Smartphones. There are now 350 Perforce users, mostly software engineers. The server and development infrastructure is NT based and there are three major development sites connected using a Virtual Private Network. The main codeline contains about 50,000 files in 1700 directories.
The author is a software architect working on software production processes at Symbian. A system manager and developer by background, he joined Psion in 1994 and became part of Symbian when it was formed in 1998. In 1999 he was part of the team responsible for introducing Perforce into the company and organising the accompanying process changes and training. Configuration management and administration are now two of his key roles.
Symbian's main product is the EPOC operating system that forms the heart of new generation Wireless Information Devices (WIDs) and may be familiar to some as the operating system that runs in Psion products.
The company's roots were originally with Psion, with development focused primarily on producing systems for their product range. This meant that development was mostly in a single stream focused on delivering specific retail products. The configuration management for this was therefore quite simple.
The company evolved into Psion Software, whose mission was to license the operating system to other manufacturers. In 1998, this took a radical step further with the formation of Symbian, a company now jointly owned by the major telephone manufacturers Nokia, Motorola, Ericsson and Matsushita together with Psion.
In this situation, the single threaded development model certainly no longer holds. Instead, the company is engaged in the delivery of several variations of its software, called DFRDs, to an increasing number of licencees. It is also growing rapidly to meet this demand and engaged in an effort to become more process mature. It was time to review the entire system of production.
The existing SCM tools and processes were based around a 1990 version of PVCS, which was used in conjunction with a set of command procedure wrappers to provide a standardised environment for SCM and for building the software. The desire to update these tools had been growing for some time, but efforts were so intensely focused on actual software development that this remained under-resourced until late in 1998 when two things started to happen. One was tangible evidence of increased complexity in software production which was straining the capabilities of the existing processes and the other was the desire for "Y2K compliance", and the existing tools could not easily be certified. So a working group was tasked with choosing and implementing a new SCM tool set.
At this time, we had about 100 software developers and the main source tree consisted of about 13,000 files in 200 distinct component directories. This was only about 120MB of data.
The main areas of interest in our selection process were:
We were aware that SCM tools come in differing classes of functionality [1]. Although some of the sophisticated, highly integrated, systems seemed interesting, we knew that we did not have the resources to conduct a proper evaluation at that level. Our primary need was for an SCM system and we confined ourselves to that. This means that one of our requirements for other development support software such as defect tracking systems is now that it should be able to interwork with Perforce!
The process of choosing a tool started with the discussions from which the above areas of interest emerged. Many of these turned into hard requirements and we used these to aggressively filter the list of possible candidate systems down to a shortlist of 5. These 5 systems were then put through a more thorough evaluation. Responsibility for each evaluation was delegated to specific individuals, who would immerse themselves in the candidate system and complete a checklist describing how the tool measured up to the requirements.
From this evaluation, the project team then chose a set of 3 tools which would be piloted by using them concurrently with the development process. This would also involve further usability evaluations. At the end of the pilot phase, none of the three tools had been explicitly ruled out. The final evaluation was therefore on the basis of perceived quality and also minimising the impact of change. In the end, Perforce was chosen because it appeared to offer support for a richer style of working than the other tools. To paraphrase one of our reviewers, it is more like an aeroplane than a car. It was also a product that was well liked by its most important potential users in the company - the software engineers themselves. This has been a factor in their continued support for our SCM processes.
The development processes in use prior to the introduction of Perforce were based on a concept of components. EPOC was divided into about 150 software modules. Each of these components was developed by a small group of engineers in a relatively self-contained environment. The formal process involved in publishing such work was called the component release. During this process the component would undergo substantial unit testing and the release would be accompanied by comprehensive release notes.
The component release would include built binaries and a complete system could be constructed by selecting the binaries from the component releases deemed to make up the configuration. The whole system would be integrated and tested, with defect fixes implemented as further component releases.
This process had the following advantages and disadvantages:
|
Advantages |
Disadvantages |
|
Unit testing and release process minimises defects at an early stage in the development process. |
Large overhead attached to minor software changes. |
|
Released binaries always available for system building. |
Valid configurations not easy to determine. |
Complete system builds were a rarity in these circumstances, though they were always part of the final release process for the whole system. Component developers would often build and release their components against quite old configurations. This was partly a measure to protect binary compatibility, but would sometimes mean that the results of building the component during an entire system build would be different from those of the normal component release.
We could see that as our deliverable requirements got more complex, the overhead of unit testing during component release would become even greater. Also we were concerned about the inconsistencies between the component build and test environment and the environment of a full configuration.
Pervasive changes - those involving work on a large number of components at once - were also a significant worry. To carry such work through to completion meant a "baton passing" exercise between component maintainers who would have to release components in sequence in order to provide a working framework for efforts higher up the dependency chain.
When introducing Perforce we had two choices:
We chose option (b) because we felt that it would give us rapid return on the investment in the new tool. We also felt that it might be difficult to carry out a separate process improvement project later - the momentum for change was building up and we didn't want to dissipate it.
For our new system, we took heed of the Best Practices [2] and the Perforce life-cycle model [3] documents on the Perforce web site and homed in on a development-main-release codeline approach.
In this model, code changes are made in development branches and integrated to a main codeline when they are working. Mainline code is regularly built and tested and represents the leading edge of development. Code is released through a release codeline which is more tightly controlled and tested. The policies of the mainline are aimed at rapid development, the policies of the release codelines are aimed at stability and quality. Code changes generally migrate from development, through main to release. Each of these codelines are associated with processes of increasing rigour, designed to improve product quality so it becomes ready for shipping.
Introducing a new development process involved quite a lot of intellectual effort - the concepts seem simple enough, but forming them into detailed processes that everyone accepts is not so simple. Even amongst those of us thinking about it, misunderstandings could occur. The challenge was to explain them well enough to the development community for them to adopt them without difficulty.
Our intent was to allow developers to attend to the core business of developing code with minimum overhead, as we perceived that the component release process was becoming stifling. The important unit and integration testing would be moved to a later part of the software production process.
To do this we focused on the key process: Submission to mainline.
The EPOC depot, where we hold our development code, is organised at the top level into three directories that represent the important codelines in the process:
//EPOC/release/... //EPOC/main/... //EPOC/development/...
The main codeline represents the definitive leading edge of EPOC development and is the focus of a lot of engineering activity. The entire EPOC system from the mainline is built on a daily basis. (See [4] for reasons to do this) This removes some of the problems we had with the component release process and also provides an important criterion for mainline code submission - the build must not be broken. To submit changes to the mainline, an engineer places them in a pending changelist which must contain adequate documentation of the work. The engineer must then satisfy a person called the build coordinator that the submission has been tested to verify that the build will continue to work and that the system produced is at least superficially viable. In contrast to component releases, part of this means that the submitting engineer must have taken care of any consequential changes to the system that may be required to interface with the specific specialist changes that they have made.
The build coordinator is a key role in this process. Without this human element we felt that the process would not be successful. We have about 15 build coordinators, drawn from the development community itself, who work in rotation. The build coordinator of the day begins by publishing the day's timetable and any variations on the submission process. At the end of the day, the coordinator publishes the accepted submissions and starts the build process. On the following day, the coordinator examines the build logs, carries out smoke tests on the system and publishes the build report. If a build fails, the coordinator identifies the submission(s) that caused it. The engineer(s) responsible are then said to be on the hook - their most urgent task for the day is to repair the system.
The mainline submission rules mean that all but the most trivial changes are probably carried out and tested using a separate codeline under the //EPOC/development/... directory. We only have very loose guidelines about the directory structure under //EPOC/development/... - development teams use whatever structures suit them best.
Formal releases of any sort are never made from the mainline. The mainline provides a vehicle for development only. Code that is to be delivered to any sort of customer comes from a Release branch.
The process of submitting code to a release branch changes depending on the position of the release project in its life cycle. At early stages of the project, a release branch may be nothing more than an integration from the mainline. At later stages, where stability and change control become important, the submission rules to a release branch become much more rigorous.
An important submission rule for a release branch is that any submissions which are defect fixes must make a statement about how the change will propogate back to the mainline - we don't like to fix a defect more than once.
In general, the model is that code changes migrate from development, to main, to release and quality increases. But at no time is the mainline closed for submissions or frozen in any way. In contrast, release branches are subject to increasingly rigorous change control leading to the stability required for a formal release to be made. This will involve various types of code freeze such that some submissions are refused. The changes involved in such circumstances will always be applied in some way to the mainline so that they appear in some future release.
With some idea of the new development model in place, we had to find a way of switching to it without disrupting development itself. We did this by introducing the Perforce service as a back end and using an import mechanism for it to acquire code changes from the world of PVCS. This meant that developers could make a transition to the direct use of Perforce at their own pace. Those that were not yet ready could continue to make PVCS code submissions which would automatically propogate to a Perforce codeline.
The migration exercise was planned to fit in with the various current development projects with the schedule converging on the month of August for a great deal of activity.
In detail, the migration process was something like this:
By the beginning of the process, the EPOC source had already split into four separate codelines. Rather than using PVCS branches, these were implemented as separate copies of the PVCS repositories derived from the original one on a share called SRC_ERA. SRC_ERA contained all the EPOC code history leading up to a release called ER5. SRC_EPOC5U was created from this to foster development leading to a special release called ER5U. It would also host a number of pervasive changes that included support for new compiler tool chains and build processes. SRC_EPOC was prepared as the basis for the emerging mainline process and SRC_CRYSTAL held work specific to an EPOC DFRD project that would eventually merge with the mainline and also give rise to a Perforce release codeline called Crystal 6.0.
This was an ambitious but carefully prepared plan. Despite its complexity we were able to carry it out almost to schedule.
After the Perforce mainline had been created out of the SRC_EPOC PVCS material, the users still using PVCS could participate in the ongoing development process because we used an import branch to take it.
The import branch represented the state of the PVCS repository and we integrated the still-changing material from that to the mainline as part of the daily build process. The influx of such material waned quickly as the engineers themselves were trained and began using Perforce for themselves. Material on the import branch itself is updated using a process much like the detached working process described in [5].
Perforce is sufficiently unlike other SCM systems that a training plan is worthwhile. We decided on a dual approach. Representatives from across the development community attended the 3-day advanced training course. The intention was that these people would then be available to transfer their expertise to the rest of their teams as required. In addition, we prepared a 2 hour internal training seminar and a tutorial document. These were focused on a worked example designed to compare and contrast Perforce with PVCS and the old and new development processes. The seminar was given to groups of about 20 people at a time so that there could be plenty of interaction.
This initial training process was completed by November and was successful in supporting the transition. We continue to top up our training with a variation of the advanced course and are also developing our own, shorter, introductory training programme for engineers.
By November 1999, all software configuration management in development had migrated to the new system. The core processes were working well and people were happy with the new system. Key factors in this achievement were:
With the new system in operation, we start seeing the imperfections. Some of them operational, some which would translate into a wishlist for product enhancements. As far as operational difficulties go, we found Perforce as a company to be extremely responsive and helpful when we had difficulties. With hindsight, I'd add an evaluation of this explicitly to the requirements list.
Here's a list of the things that we've had trouble with:
All of these things are addressable in some way but they weren't things we were able to anticipate.
The most troublesome problem has been connected with synchronisations that seem to stall indefinitely. This is hard to diagnose and may actually be connected with a local network configuration problem rather than with any defect in the Perforce service itself.
Some problems we've had were traceable to known defects in the 99.1 server that are fixed in 99.2. Once identified, we were able to work round them, but the obvious solution - to upgrade to 99.2 - took a while to implement. Here is our upgrade process in overview:
Some would say that this is overkill, but our business depends on this material and we wanted to take no chances. Our difficulties with this process were:
Of course in practice the upgrade was a success and we had no need to downgrade the live system.
Symbian is a distributed company, with offices in the US, London, Cambridge, Ronneby (Sweden) and Tokyo. The Perforce servers are all in London. Perforce is designed to be a client server system and does not of itself support a distributed server structure. There can be bandwidth issues at some sites. These are ameliorated in various ways but we don't think there will ever be complete equity of service between users close to the servers and those operating over a slower connection. Here are the things we have done.
Perforce has grown in a Unix environment where the whole system is case sensitive. We use NT, whose filing system (NTFS) is not normally configured to be case sensitive. In all important aspects, the Win32 implementation of Perforce works well. However it is not perfect. For example:
P4 add myfile.txt
The file saved in the repository will have the all-lower-case name that the user typed, even if the actual file was called MyFile.TXT. An NT user would expect the actual case of the file to be honoured.
Sometimes things have happened to cause parts of the repository to change case. Again, this is largely an aesthetic rather than an operational issue, but it has caused problems for one of our cross-referencing programs which happens to have come from a case-sensitive design itself.
Perforce allows difficulties that some other systems can only dream of. Users sometimes need to remove the effects of a recent submitted changelist. Although there is published advice about this, sometimes it would be helpful if the system more explicitly supported this.
Users also sometimes expect a lot of the integration process. Tracking the integration history of a file can be a confusing business and sometimes the merge options offered are not what the user expects.
These issues probably fall into the same category as the oft-repeated request for an explicit rename command. It is all scope for improvement. However, we also know that defects emerging from such extra complexity would not be welcome!
Our db.have database is often more than 2GB in size. So we are conscious of the need to prune out the client views that are no-longer used. This can be a labour intensive process. When a user leaves, one cannot always assume that their clients are no-longer needed. We've found clients mapped to large areas of the repository (a common mistake made by new users) that just fall into disuse. No doubt we will refine our housekeeping tools with time.
As our experience with the mainline process evolved, we began to observe that development on higher levels of the system could suffer because a recent, reliable, version of the system build would not always be available for people to use as their development environment. In addressing this problem we developed a Base level process which delivers stable system builds for developers at planned intervals.
In Perforce terms what happens is that a codeline is constructed which takes its configuration from a recent good version of the mainline. This codeline is worked on as necessary to make it completely robust before a build taken from it is delivered as a working platform for developers. No further work happens on the base level delivery codeline until the next full integration for the next base level delivery. Thus the codeline itself is not a release branch in the normal sense - what it delivers has a very short life cycle and does not take on a significant life of its own.
At Symbian, the base level process contains more detail than is sensible to describe here. It includes involvement in project life cycles to allow base level deliveries to have planned functionality. The base level process is now being embedded into our project processes.
Internal projects in Symbian are planned using a standard framework and life cycle model. This now includes specific sections on configuration management so that it is well understood when, for example, the focus of a project moves from main to release codelines and what specific rules and processes are applied to the release codeline during its life.
Delivery of code to customers always goes through a release codeline. The released package consists of:
The source code that goes with a release is to allow the customer to carry out the work they need to do to turn the generic Symbian software release into something that is specific to a product such as a communicator.
The entire source package has complex origins. Much of it is originated at Symbian, but it also includes code that has come from development partners.
In delivering this code, Symbian has to be sure to honour its Intellectual Property agreements with its development partners and to protect its own interests regarding the proliferation of its own code. This is done by tagging each source directory with a policy description that is used by an automatic filtering and reporting program used in the construction of the exported source tree. The challenge here is to allow a delivered build process to continue to work after source filtering.
The release codeline at a given changelist number is the starting point for the construction of a release package. The intent is that this process is entirely reproducible from the contents of the repository - it does not rely on any other source of material.
As well as undertaking generic system development, Symbian also works directly with some licencees on their specific products. This is done by a division in Symbian called Licencee Product Development (LPD). The development model in LPD is quite complex because of the integration with licencee development processes and the need to protect the licencee's own intellectual property on site.
We decided it was safest to address the security issues by having a separate Perforce server for LPD, in which we could operate a different protection model. The use of a different server allows for a separation of administrative function. As it turned out, we ended up with centralised administration, but of course this can change as the company continues to grow and evolve.
The flow of material from development to LPD is organised something like this:
Development releases are made which deliver a binary package and appropriately filtered source. This delivery is called an OEM Construction Kit (OCK). In addition, the source is submitted to an import codeline in the LPD server. Individual licencee projects can integrate this code into their project-specific codelines for further development.
Symbian works with several development partners in production of core software and also during licencee product development as shown above. Development partners do not normally have access to the Symbian Perforce servers and may well be using their own configuration management tools. Development partner processes centre on the delivery and integration of code in both directions. This has to be accomplished with minimum overhead and complexity.
The most important factor here is not so much the SCM technology itself, but the quality of the planning and communication that goes into the development partner relationship. A "throw code over the wall" mentality will not achieve good results no matter how good the SCM tools.
A common pattern is for Symbian developers to act in proxy for the external partners. The process looks like this:
In this diagram the codeline at the top represents a consolidating codeline from which deliveries may be made to external organisations. A normal development pattern is for individual developers to maintain private codelines from which they integrate their work back to a common place. The diagram illustrates how this work takes place at each developer's own pace. The developer called Stan is in a remote organisation. Someone locally maintains a codeline for him and ensures it is updated with an integration from the relevant external code delivery. The returning code is imported into Stan's codeline and then integrated back into the common stream.
The mechanism for importing the changes is similar to the detached working procedures described in [5]. The level of integration done offsite depends on the project. The individual developer codelines shown allow remote workers to deliver code at their own pace if required, without first having to resynchronise with a later delivery from Symbian. An alternative approach might be to have the external organisation carry out its own consolidation first. The viability of this depends on the degree of entanglement between code being worked on externally and internally.
Symbian is also experimenting with controlled access for development partners to external Perforce servers shared with internal developers. Code from the external server then has to be integrated back into the main internal server codelines, however early indications are that this method of working provides a big improvement in turnround compared with the proxy integration processes.
We suspect there is still a lot of scope for improvements in this area for us.
The main Perforce servers are run by our IS department. Perforce administration itself is carried out by a team of six people - two in IS, two development staff with specific CM responsibilities and two other development staff (drawn from the build and integration team) to provide operational redundancy. This cross-functional team works very well.
The servers themselves are dual processor 400MHz Intel NT machines with RAID 5 storage. In use, CPU utilisation is normally between 5 and 30 percent. The majority of clients are also NT machines.
The intent is to provide 24 hour operational cover, though users see a service delay while the checkpoint dumps are running. We currently do these daily at about 2300, but as they take about 45 minutes to produce a 3GB checkpoint file we will review this strategy. The Perforce database files themselves, particularly db.have, do tend to grow and we restore these from checkpoint from time to time.
In organising the repositories, we chose to use the depot concept to embody high level policies. For example, the //EPOC depot embodies all the policies (and more) that I've described above relating to development. There are other depots serving different user communities, each of which has its own policy statement. At the bottom end of the scale, each server has a depot called //PLAYPEN in which users may experiment, the only rule being that it must not be used to store confidential material or anything that is being depended on as part of Symbian's work.
The protection model is kept as simple as possible. Protections are organised so that no user can obtain access to a server unless they are a member of a Perforce group. We use a group called "authorised_users" to hold all usernames with some kind of access. Particular depots are then controlled with their own groups, for example, members of "EPOC_writers" have access to the EPOC depot. We try to keep to an ideal which is that people who aren't potential writers to a depot have no access at all - they should be obtaining material in its published forms. In practice there are always exceptions and these go in groups such as "EPOC_readers".
We also publish clear policies about the use of Perforce so that users are aware of their responsibilities such as setting passwords.
Authorisation of a user is carried out by a simple NT command procedure which validates the username and updates the appropriate Perforce groups. We maintain an online written log of all administrative actions which keeps administrators informed of each others activities and is also useful has a historical reference. Although administrative actions appear in the Perforce journals, this information is not easily accessible for reference and there is no opportunity for commentary on it in the Perforce metadata. This might by an area in which we'd ask for improvements from Perforce.
Configuration management of the material used directly in software development is well understood. We are less clear about how the scope of our CM system should be broadened to include other material that is relevant to the management of software production and the general running of our business. Examples of such material are:
All this material interacts in some way with the software production process. It is good to model the larger configuration because it allows us to manage the business better. The issues facing us are:
I'm hoping that we'll have the answers to some of these questions during our next year with Perforce.
Further information about some of the terms mentioned above can be found via the Symbian web site at www.symbian.com.