The Hardware Design Challenge

by Shiv Sikand

Abstract

Implementing software configuration management (SCM) practices in a hardware design environment is a complex task. While hardware design shares many aspects of software design, particularly hardware description languages, the development model is very different to that of software design. The added complexity of physical design databases, usually in proprietary binary formats diverges the use model even further. However, the unique design of Perforce can be used to easily facilitate software configuration techniques to manage the entire hardware design process. The rewards of adopting such a flow are greater accuracy, control and productivity with significantly reduced administration overhead.

Historical data management methods

Traditionally, chip design managers have used some combination of TCM - tar, copy, move to manage their design trees. This process usually involves a database ‘czar’ who spends large amounts of time ensuring the consistency of the design tree in its various stages of development and ensures that the different disciplines such as logic design, circuit design, verification and physical design all have the correct versions of data and that the final database correctly integrates all the required changes to guarantee the correct functionality of the manufactured part.

TCM activity usually follows a fairly well defined method - a ‘freeze’, followed by a period of intense activity where all the different groups work to synchronise their individual changes into a common, working database that accurately represents the current state. Such custom built TCM systems are typically highly effective, but are expensive to maintain and run. The difficulty of managing such systems is compounded by the requirement to manage multiple variants of the same database to meet specific schedules and/or features.

‘Freeze’ based methods also slow down the pace of development since they are not incremental. A new set of copies is typically required to create a new database and is often error prone due to poor management of the underlying data by the users themselves.

Derived data issues

As the design progresses towards tape out, we see an exponential growth in the size of data files. These typically take the form of change dump files, analog simulation output and parasitic data. While this data is generated, the generation time is long. In addition, tools that generate the data may change versions and have slight incompatibilities often making it impossible to reproduce the exact behaviour of an older design. Tracking and managing this data is a very important part of the whole chip configuration problem.

Chip variant management

For any given design, there will probably be a large number of variants. The reasons for these variants include process issues, system bugs, timing and crosstalk problems, and changed customer requirements. Propagating the right set of changes to the appropriate variant is a very difficult problem without a powerful SCM system. In many cases, the work is repeated across multiple variants since there is no easy way to apply sets of changes to snapshot or freeze data. This is extremely inefficient and error prone and is often the cause of wasted mask sets in the fab.

Common libraries

As companies tend to produce more than one chip, many components are shared. Managing these shared components is a time consuming task since the producers of the data have enough to worry about without worrying about where the data will be used. By implementing a development model that enables automatic change propagation, sharing, reuse and collaboration are optimised.

Multiple fabrication foundries

Many designs are multiply sourced-out to different foundries. Without powerful SCM tools, replication of data sets is unavoidable and adds an extra degree of complexity to chip variant management when it comes to change propagation.

 

The Perforce Solution

Differences from SCCS based archivers

Perforce is a powerful SCM system rather than a simple revision control system, or archiver such as RCS or CVS or other SCCS based archivers. SCM is the ability to manage sets of files and their associated dependencies automatically. Many traditional tools provide configuration management through the use of symbolic names or tags. This explicit configuration style is in direct contrast to Perforce’s implicit configuration management abilities using its powerful relational database.

Change based architecture

Every transaction in the system is assigned a unique change number, making it very easy to track changes throughout the entire tree. As state is stored centrally in the server and not in the workspaces, data recovery, mining and reporting tasks are fast and easy to perform. As Perforce follows a client-server architecture and metadata is stored on the server, all state can be queried directly through the database rather than having to collect file status information from distributed workspaces across your network.

Atomic transactions

Atomic transactions allow sets of files to be tracked together. The key to a good SCM system is its ability to track change packages and the ease by which these change packages can be propagated. Even though each file in a codeline has its revision history, each revision in its history is only useful in the context of a set of related files. The question "What other source files were changed along with this particular change to foo.c?" can't be answered unless you track change packages, or sets of files related by a logical change. Change packages, not individual file changes, are the visible manifestation of software (and by extension, hardware) development.

Inter-File Branching

IFB is the cornerstone of working with configurations. It begins with the natural model of copying software files and renaming them - and finishes with a collection of techniques that make its model usable.

In practice, if branching a file means copying it in the repository then storage space can be a concern. A simple antidote is to perform a virtual copy, where the newly branched file makes use of the contents of the original file. This requires a level of indirection between the repository name space and the actual underlying object store. When the branch is extended by adding a new revision, the branched file can acquire its own separate entity in the object store.

Supporting variants with virtual copies reduces the space requirement of a variant to be merely a record for the newly created variant that points to the original file. This virtual copy can also be used when one variant is explicitly synchronised with another. If the user wishes to fold a branch back into a trunk and make the two identical, both can reference the same content at that point.

Incremental abilities of IFB

The change from virtual copy to entity is tracked internally and provides a complete audit trail and can make for comprehensive reporting. Through transitive closure, it is possible to compute for any revision of any file whatever deltas it incorporates from other files through first, second, third, etc. generation merges or replaces.

This powerful feature allows changes between branches to be propagated automatically. There is no need for diffs (often an unusable technique for determining changes on binary objects), or other manual techniques to determine the difference between trees that shared a common fork point. It works seamlessly across design hierarchies too, because Perforce is able to compute and apply the incremental updates between data sets and also report them with ease.

Tags versus IFB branches

Tags are generally very inefficient in comparison to IFB techniques for a number of reasons:

• They are not incremental, so data must be continually re-tagged
• It is difficult to visualise or get reporting information on the difference between two or more sets of tags
• They proliferate and make tree management more complex than it needs to be.
• It is a generally slow operation in most databases, both for the creation and recovery of file sets
• It is an explicit operation - if you didn't tag something, it is next to impossible to return to that state. With Perforce all state is recoverable at all times. This advantage cannot be over-stated
• In Cadence and other CAD databases traversing the hierarchy using Skill, Scheme or other languages to build a tag set is an inefficient way to map data sets and requires the database to be active in order to make a configuration.

Implementation strategies

Perforce implementations will vary according to the specific nature of the chips being designed. For 100% synthesis based ASICs, the majority of configuration management activity will take place at the RTL level. In general, for RTL and other hardware databases, branching and merging is not the normal activity as it is in the software world.

If Place and Route is performed in-house, the Avanti Milkyway database is extremely amenable to Perforce revisioning. For other chips, where there is a component of custom layout and schematic databases, the Open Source Cadence-Perforce integration can manage your Cadence layout and/or schematic databases seamlessly. This integration is coded in less than 5,000 lines of Skill code and provides Cadence users with the power of Perforce in an integrated GUI. It has the richest feature set and best performance of any known Cadence integration including TDM, DesignSync or ClioSoft for zero additional capital cost once Perforce has been licensed. The server hardware requirements can be met with low cost PC hardware running Linux or FreeBSD unlike other systems that require expensive proprietary machines.

Branching strategies

There are three specific models for branch and replace for hardware design, namely shared, decoupled and independent.

In a shared branch model, users have a common tree that is branched to form part of a specific revision of the chip in question. Each user has their own independent workspace and they can either manually synchronise with the tree, or use an auto-update job that runs at a fixed interval to keep their tree synchronised.

The shared branch has a specific advantage - checkins are instantly visible, allowing advanced collaboration. The integrations between the shared branch and other parts of the design tree can be handled by an integration owner, simplifying the use model for users and establishing a formal integration policy.

For new designs, the hurdle any SCM system must overcome is the ability to create a composite view as a slice through the developments threads of different data sets. In many cases, the maintainers of a particular data set have enough to worry about dealing with just that set’s evolution. It is too much to also deal with how their data gets used by the different projects, hence the requirement for an integration methodology that is not always visible to every user.

In a decoupled model, multiple branches of the same data set exist. These branches may represent different partitions of the design as required and have a slightly different temporal distribution. In this model, checkins are not visible to users outside each individual branch until after an integration. The integration is handled locally with the help of a defined set of integration policies. A simple policy example is to disallow stale data propagation back to the parent. In other words, the branch must be fully forward integrated with respect to its parent before data can be reverse integrated. Since Perforce tracks the parentage of all objects, this is accomplished very easily. A decoupled branch may also require the use of exclusive checkouts across a specific named set of branches. Again, the powerful relational queries available in Perforce makes this a trivial task without the use of complex administration procedures. The Cadence-Perforce integration, for example, supports both shared and decoupled strategies and can guarantee exclusive checkouts as required.

In the independent model, branches may follow their own evolution without respect to other similar data sets. This model is particularly valuable for concurrent design where the outcome or result of an experiment or design change cannot necessarily be determined in advance. It is the perfect method for exploring multiple ‘what if’ scenarios in an extremely short period of time, since any changes in the main trunk can be quickly and automatically propagated to the independent branches. This is another example of the extremely powerful incremental abilities of IFB and found in no other SCM tool today.

None of these models are mutually exclusive, in fact they are entirely complementary and can morph from one style to another at any point in their evolution. This ultimate flexibility enables rapid turn around time of new designs and ideas without recourse to long and potentially very costly database freezes required in a conventional TCM model. As the design progresses to tape-out, the requirement to guarantee the correctness of configurations increases dramatically as does the data set size. Perforce’s IFB model effectively eliminates this bottleneck and can allow very quick turnaround of new ideas and processes late in the development cycle while ensuring accuracy and configuration integrity.

Implementation examples

Take, for example, the design of a standard cell library. This development occurs in a shared branch, with each user working on the library having their own copy of the data. They achieve visibility into one another’s work by either manually synchronising their data, or by the use of an auto-sync process. They check in changes regularly until they achieve a point of known goodness i.e. the set is DRC and LVS clean and ready for testing by the next team at which point they ‘release’ their data by integrating into the release tree. Since Perforce understands the dependencies between the two trees, it will only propagate data that has changed between the two sets, resulting in an incremental update. The p4 query commands allow everyone to articulate the nature of the change and provide the necessary design communication between producer and consumer. Since the integrated changes are propagated atomically, data integrity is assured. In traditional tagging operations, you can never be assured that the entire data set was propagated correctly and since tags are not incremental, the operation is time consuming and error prone. Many people re-use tags to avoid this situation, but that in itself is a risky operation and violates the first principle of any SCM system, as it denies one the ability to return to the state specified by the original tag contents.

By adding additional branching layers, it is possible to have very fine-grained control of your development work since it allows you to decouple and parallelise your work. Suppose the last release of the standard cell library changed the routing pitch that impacted a consumer negatively, or caused VDD/VSS shorts. The consumer has the option of reverting back to the previous release because they control the updates to their workspaces rather than relying on the producer to fix the problem immediately. They are both able to continue their development and re-synchronise at some later point. In TCM systems and conventional SCCS based systems, this decoupling is difficult, and the incremental features virtually impossible to obtain.

In addition, parallelism is easy. In this example, Foundry_experimentxx contains a set of optimisations that you may want to explore. By creating a new branch for the experiments, it is possible to work on alternative approaches and then re-synchronise changes between the dev tree and experiment tree very easily. The arrows show the movement of only the changed data, rather than the entire data set. This is a big win in speed and filesystem size. The release candidate can then be chosen from either tree. The consumers of that data in the release tree always know the source of their data by the use of simple p4 queries.

Figure 1: Decoupled branches

Figure 2: Project assembly

Rather than a tag hierarchy that only makes sense to a few people, the project configuration can be expressed as a collection of branches. In most other systems, it is almost impossible to determine the set of tags used to assemble the particular state of a design without the use of some external application such as a web info page that is manually updated by the people building the configurations. The Perforce configuration is fast, error free and completely incremental. It is very easy to change the configuration state of the project by modifying the branch references rather than by re-syncing to different tags or by copying large data sets. The potential impact of changes can be easily visualised or tested, significantly reducing turn around time for design issues and engineering change orders. This type of structure is very easily extensible to match the specific requirements of a particular chip variant. Let us assume for example that a special IO pad is needed by rev3.1 of mychip. This IO pad is actually a foundry_release object, but is locally modified. Often, these modifications may be chip variant specific, or they may be as simple as being tool specific - e.g. it may have a special texting requirement to ensure a DRC waiver.

The configuration can the be easily modified by branching the foundry_release tree directly into a chip specific variant.

Figure 3: Multi level decoupling

While this example is contrived, it shows how complex dependencies can be managed through the use of IFB, since Perforce’s integration history automatically tracks the changes between branches. The chip integration manager, or the designers themselves, can control what data is used to assemble their design. There is no risk of stale data or data of unknown origin being used, as all data propagates through the Perforce database and not through copies of data out of individual user’s workspaces. By reducing the number of copies and managing their distribution automatically, complexity and database management time is reduced. The human bottleneck in the form of the database czars can often be removed entirely from the equation because their job is now oversight rather than an entirely mechanical co-ordination and copying process.

Tool Integration

Third party tool integration can often be a daunting task. From a developer’s perspective, linking a tool to an SCM system is best achieved through a trigger architecture. In such an architecture, the tool provides a hook for all file-based activity. A good example is the Skill trigger model for Cadence DFII. In this example, a pre and post trigger function can be registered for all file activity that requires state changes capture. Examples are object creation, object deletion, checkin and checkout actions. It is also very important to engineer a good scheduler to handle error conditions for each trigger, rather than simply implementing pass/fail behaviour for the triggers.

From the user’s point of view, integrating an existing tool with no trigger model can be performed using scripting languages such as Perl and Tcl/Tk. This method is particularly powerful if a set of commands exists to launch a design from a scripting shell. It is a good idea to use IPC or other message passing protocols that may be available to avoid paying the fork penalty for every Perforce command. Using the P4API set of utilities, it is possible to build your own Perforce client with the required characteristics. A good example of this can be found in the ‘p4skillserver’ tree of the Cadence-Perforce integration in the Perforce Public Depot.

It is also important to provide the simplest level of interaction with the tool, namely low-level object interactions. Complex activities such as branching and integration policies should be handled outside of the tool integration to avoid unnecessary complexity.

Perforce in the real world

The following data is from an example installation of a small, full custom, fabless semiconductor company. The Perforce database had been active for less than a year in this specific example at time of writing. The database contains schematics, layout, place and route, verification and RTL data.

Number of branches		68
Number of changes		85,987
Total number of files (at head revision)		2,675,545
Total number of users		36
Total number of clients		3,219
Perforce depot size		15 GB

Creating a high performance Perforce installation

Due to the large numbers of files and their sizes in a hardware database, it is very important to optimise the Perforce server hardware. In such a situation, I/O bandwidth and filesystem performance is critical. The depot filesystem must be local to the machine. There is a growing trend with the advent of Network Attached Storage (NAS) to use these machines as part of the Perforce server. However, due to the poor write performance of RAID4 or RAID5 subsystems found on such hardware, it cannot be recommended.

The best price/performance is achieved on hardware RAID 0+1 SCSI systems running on Linux x86 hardware using the SGI XFS advanced journaling filesystem. This feature is available in the 2.4 kernels and is standard in Mandrake 8.1 and Red Hat 7.2 onwards. Linux also supports asynchronous directory updates which can make a significant performance improvement when branching. On Solaris machines, inode creation is very slow and can result in very long iowait states.

Alternatively, SGI IRIX can be used since its native filesystem is XFS but the price/performance point is less favourable. In addition to file system performance, the journalling ability of XFS is critical to system recovery times after crashes or power failures. This avoids the need for ‘fsck’ consistency checking after a failure which can result in significant downtime as the database grows in size.

Managing depot size

By definition, SCM systems are grow-only databases. The use of ‘p4 obliterate’ to prune data sets is not recommended since it irrevocably deletes data from the depot. Additionally, if you are using the Cadence-Perforce integration, or other system that relies on branch integration history to correctly manage data, a ‘p4 obliterate’ will result in errors.

The best way to prune your archive is to leave the meta-data alone under all circumstances and archive the respective depot directories to tape, or other backup medium. This will allow all integration history to be reported correctly. Should an archive file be required, Perforce will return a ‘Librarian Checkout Error’ when an archived file can longer be found. These errors can then be formatted into a backup recovery request if required.

Summary

Perforce’s unique Inter-File Branching mechanism is a powerful way to manage the hardware design process. The high performance Perforce client-server architecture is ideally suited to large file size and binary databases. Its low cost, low hardware requirements, and minimal system administration cost make it easy to adopt across the enterprise.

The key of a good configuration management system is its ability to track change packages and the ease with which these change packages can be propagated. Perforce’s unique feature set and underlying architecture presents a way to implement SCM practices in a hardware design environment, which can result in greater accuracy, control, and productivity with significantly reduced administration overhead.

About the Perforce SCM System

PERFORCE is the Fast Software Configuration Management (SCM) system, architected for high performance and runs on a wide variety of platforms. Perforce provides comprehensive configuration management, including version control, workspace management, atomic change transactions and a unique and powerful branching model. Perforce has been designed to be easy and intuitive to use with a minimal administration overhead.

About Perforce Software Inc.

Perforce Software Inc. designs, develops, markets and supports Perforce, the Fast SCM System. Headquartered in Alameda, Calif., Perforce Software sells worldwide and has international operations in Europe, Japan, and India. Perforce customers include developers of systems, application, Web, and database software; and firms in the financial, education, and research markets. For additional information, contact Perforce via e-mail at info@perforce.com, the Web at www.perforce.com or phone at +44 (0) 845 345 0116.

About the Author

Shiv Sikand received his BSc and MSc degrees in Physics and Electrical Engineering from the University of Manchester Institute of Science and Technology. His main interests are in design automation and verification techniques for integrated circuits as well as Open Source software development.

After completing a post-graduate research position designing asynchronous chips in the AMULET group at the Department of Computer Science at the University of Manchester, he joined Silicon Graphics in Mountain View, California where he worked on high end MIPS microprocessors and advanced graphics chips for more than 5 years. After a brief foray with network switching fabric devices, he is now at Matrix Semiconductor, Inc. in Santa Clara, Calif. working on the world's first 3-D memory chips. He can be reached via email at shiv@sikand.org.