Integrated Software Configuration Management
for Hardware Design

Abstract

Software Configuration Management (SCM) has traditionally not been available in the hardware design world due to the use of proprietary tools for design capture which were not flexible enough to offer integration with commonly available SCM systems.

Recently, Cadence Design Systems has made this integration possible with their design capture tools. This paper describes the design and implementation of a high performance scalable SCM system for hardware design.

Introduction

SCM techniques are well defined for software development efforts. Hardware design has a different set of requirements mainly because of the mix of both ASCII and binary data that eventually defines the physical mask layout of an integrated circuit.

Design capture for integrated circuits (ICs) usually consists of four aspects:

• System level specifications in either C/C++ or proprietary language
• Register Transfer Level (RTL) specification in a Hardware Design Language (HDL) such as Verilog or VHDL
• Schematic capture using Cadence Composer
• Layout capture using Cadence Virtuoso.

Revision control has traditionally been available in all four areas; however, an integrated solution permitting true software SCM methodologies has not been possible due to the lack of a common tool set.

This paper will describe an SCM environment designed and built by SGI to facilitate the production of its next generation multi chip advanced graphics product (codenamed Bali). This system does not incorporate the system level specification. Instead, it focuses on providing a highly productive environment for logic designers, mask layout designers and CAD engineers.

The Design Environment

SCM for IC design needs to offer a unique set of features: the ability to handle extremely large databases, a rich feature set that supports easy integration with third party CAD tools, highly scalable performance and ease of use. Perforce is the only real choice for this solution, particularly on account of its scalability, which will be described in detail later.

In a hardware environment, data often needs to remain in a known state for long periods of time since the operations on that data are very complex and require detailed investigation and verification. However, the ability to refresh that data from a fast changing baseline, as well as the need to re-integrate results from the verification process, is critical. Traditionally, this has been a highly labor intensive and error prone task, consuming a lot of unnecessary resources.

Perforce's Inter-File Branching Mechanism [Seiwald96] is the clear solution to the problem of creating and maintaining these design snapshots. Because of its ability to track and report dependencies through an integration record, automatic, incremental updates to snapshots are at last available to the hardware designer.

The branch is used to create an insulated data set on which back end tools can run. If the tools modify the data in any way, the changes can be reverse integrated into the mainline. Eventually, the last branch will contain the data set from which the chip is fabricated. Since most of the changes are to binary files, merging is not a major issue and the data is simply replaced.

The Use Model

We will consider the interaction between logical and physical data here, since the branching model for pure software development is already well understood [Wing98],[Vance98]. These software models are used in conjunction with the hardware design model to provide a complete solution to the software and hardware aspects of the design.

In addition to foundry-supplied libraries and simulation models, the design tree typically contains a large number of components that describe in detail each aspect of the chip's functionality. The tree is usually very deep and has a vast number of files with complex dependencies. These dependencies are extremely hard to track and as a result, snapshots tend to be discarded very often, mostly through unforeseen interactions and human error.

The logical functionality of the chip must remain stable so that the physical implementation can take place. The physical implementation usually takes an order of magnitude longer to complete, so the logical functionality must be able to proceed at its own pace, with the ability to update the stable data at any time. A Perforce branch is a simple and elegant solution to this problem.

Conceptually, the tree is usually split four ways and we map those boundaries into multiple clients. This is a shared tree that everyone references, but does not edit directly, except for the Cadence libraries. This violates some SCM fundamentals for software which don't necessarily apply to hardware databases, since multiple edits of the same cell are not permitted.

The first client controls the logic side, the second control the tools, the third controls the data for the target foundries and a fourth set of clients control the binary CAD tool databases. This allows the appropriate sub-section of the tree to be synchronized independently.

Simple branching

In our current design the chosen level of granularity for a branch is an entire chip. This will vary according to the size and complexity of future designs.

The overall effect of a branch is to produce a set of managed, multiple trees with a complete audit trail. Multiple trees bring with them their own set of problems but these can be mitigated by referencing the roots through an environment variable. All data references and tool execution paths are indirected through this variable, named $PROJECT in our system.

The chip branch specifications are controlled through the use of dummy clients, which are used as templates for the creation of the branches and their clients. The branch name is either automatically created with a date specification or can be user specified.

All edits specific to the chip typically occur in the branch. Library development follows its own branch path to provide the necessary insulation.

Additional issues associated with the use of variable tree roots are as follows:

• Execution paths
• Shared dynamic libraries
• Hard coded path references in tool configuration files

Execution paths are handled by a set of links and start-up scripts for tools in the bin directory of the tree. A .cshrc file automatically maps the correct $PROJECT/bin into the user's path when it is sourced.

Shared libraries are handled by the use of the LD_LIBRARY_PATH variable, which in Unix defines the location of the shared objects and is also automatically managed through $PROJECT and its .cshrc file.

Hard coded path references are slightly more complex and use an auto expand scheme. This scheme requires the generation of a matching file, which instead of containing the absolute path, now contains a path referenced through the environment variable that defines the root of the tree.

The expand environment builds a set of Jam/make rules which executes a Perl script to replace all references to the environment variable for each branch. The non-expanded version is thus the source object and changes can safely be pushed to branches. This functionality is implemented through a Perforce trigger.

Changes made in the branches can be reverse integrated into the main line. Blocks that have successfully made it through a placement, routing and verification process appear in the main line as completed objects. The next time a branch is created, the completed blocks are automatically part of the new data set. This approach has considerable value, since only the changed data as identified by the Perforce integration record makes it back into the main line or vice-versa, rather than the traditional method of blind copying. In addition, the audit trail is invaluable in detecting and subsequently correcting errors.

Bug fixes to tools or logic netlists are pushed to branches through a simple administrative interface which controls forward integration to all or specific branches. Since all updates to the branches are logged transactions, restoring the state of a branch due to bad data is a simple operation. Branches can also be snapped from a given change number which allows main line corruption to be easily reverted.

Complex branching

We also use a slightly more complex branching scheme, known as a BoB, or branch of a branch. Often, a branch has been active for a time, and the designers want to conduct experiments or test new tools on the data without compromising the branch's integrity since they have often spent considerable effort in getting the data to a particular state.

The algorithm that defines a BoB is as follows:

• Get the versions of the files in the branch
• If they have been locally edited and reverse integrated, branch the corresponding mainline version into the BoB
• If they have been locally edited, but not reverse integrated, branch directly from the branch into the BoB

A BoB parent can thus be any branch which is very useful.

The BoB building is accomplished through the use of temporary labels. This is the only use of labels in our entire system, since a primary objective is to build a purely change number driven system to prevent label proliferation that has plagued us in the past with tools like CVS and ClearCase.

The Cadence interface

The Cadence interface to Perforce-developed at SGI, is the key to this system [Sikand98]. Since the Cadence database representation consists of co-managed files to represent a design, or cell, we use the atomic check-in feature of Perforce to encapsulate the co-managed set for any given cell. This approach has both pros and cons. The advantage is that the cells are managed directly rather than having a layer of indirection which requests individual checkouts or check-ins for a cell, which results in a very high performance solution. The disadvantage is that the atomic check-in feature is now limited to a single cell, i.e., a set of changes within a library can no longer be atomic, but is instead a set of check-ins. However, in practice, this does not impose a limitation other than a high rate of consumption of change numbers, since a cell is a natural change atom rather than a library. Given that we are running a 64-bit system, there is no shortage of headroom for change numbers.

Each Cadence library has its own client. Here we violate a fundamental assumption made by most software SCM systems, namely the sharing of workspaces. This was an intentional and calculated decision. The interface does not allow multiple checkouts of the same file by multiple designers so the merge problem is eliminated. This allows all library transactions to be centralized which makes for easy backup, restoration, disk space allocation, and provides us with the ability to view changes as they are made, rather than wait for check-ins and repeated library synchronization.

Since the libraries are typically edited in a branch, the traditional problems of live edits in a common workspace do not apply because the edit is typically not complete until the data is reverse integrated back into the mainline.

The users also have the ability to create a local workspace for the library if they so choose through a menu pick, but in practice this is a feature not used very often. Users prefer the use of the branch model to creating a workspace, since most of the time they are looking to create a stable area on which to run regressions, etc. Since most Cadence library data has complex dependencies (matching Verilog files, LVS and parasitic data), a local workspace is of limited use other than for the purpose of decentralizing the data, which may be a viable solution for the very paranoid (or NT user).

The Cadence data and any other data are now indistinguishable, which is a key concept that enables this approach to work elegantly. Native Perforce commands can be used for almost any configuration building which eliminates a dual strategy for ASCII and binary files as well as reducing the overall complexity when dealing with such issues.

A branched library is no different from a main line library. It merely has a different path and corresponding client, and so is treated no differently than any other library by the tools. Reverse integration is handled using native Perforce commands through a Skill interface. As a result, the overall performance of the Cadence interactions is very good, in marked contrast to other commercially available solutions.

We built our own library browser for Cadence so that we could provide a GUI for designers to revert versions, build version copies, query the database for checkout patterns and also automate forward and reverse integration of cells and libraries.

Custom infrastructure

The infrastructure consists of a few key components and Table 1 shows the relatively modest line counts required to achieve the integrated SCM goal.

Table 1: Low complexity

While many new features continue to evolve, the overall time spent to provide core functionality was less than 6 man months.

Scalability, Performance and Reliability

Our Perforce server runs on an SGI Origin 2000 server running 64bit IRIX 6.5. This machine has 16 processors, 8GB of main memory and 300GB of disk space and is a general compute server.

The ability for Perforce to fork multiple processes for multiple users offers significant performance advantages on an MP server.

Table 2: Data sizes (as of May 1999)

Table 3: Branch data

Table 4: Perforce server performance (/bin/time, seconds)

Table 5: Reliability

We experienced a major filesystem failure when growing a partition to add an extra disk due to a hardware failure. A large number of files were lost. However, the metadata was not damaged so we were able to use the diff of the changes between the damaged data and the recovered data to produce a list of lost changes from the backup point to the time of the database corruption. This enabled us to guarantee the integrity of the data, which was a major win and would not have been possible with many other SCM tools.

The filesystem started to have problems on a Friday evening, but we did not suspend the server until Monday morning. We then recovered the filesystem from the Thursday night incremental back-up. The productivity loss was approximately one week and would have been greater had we not had the metadata as a reference. We now have a mechanism in place that shuts down the server if the `p4 verify' command generates BAD or MISSING messages. This is run by cron every hour. Had we had such a mechanism in place prior to the crash, we would have prevented the check-ins over the weekend and early Monday which complicated the recovery task. Additionally, we now checkpoint and journal the database to a different filesystem twice daily.

Conclusion

The underlying architecture and design of Perforce is clearly suited to large databases and its rich feature set makes powerful SCM techniques easily available for hardware design. The combination of Perforce software and SGI Origin 2000 hardware provides a very high performance, high reliability solution.

References

[Seiwald96] Christopher Seiwald, Inter File Branching White Paper, Perforce, March 1996

[Sikand98] Shiv Sikand, Cadence-Perforce Integration, Perforce `98 User Conference, Oakland, CA

[Vance98] Stephen Vance, Advanced SCM Branching Strategies, Perforce `98 User Conference, Oakland, CA

[Wing98] Laura Wingerd and Christopher Seiwald, High-level Best Practices in Software Configuration Management, Eighth International Conference on Software Configuration Management, Brussels, July 1998