2001 User Conference

This revision contains revised information which became available after the revision printed for the hard-copy proceedings of the User Conference.

An earlier revision recommended the use of the -llock mount option for Solaris Perforce servers with data on NetApp filers.

Recent feedback from such sites has cast doubt upon the reliability of this configuration.

I am now, therefore, recommending, for sites running Solaris Perforce servers with storage on NetApp filers, that:

• The -llock should not be used with Perforce, until further testing can be performed. Specifically, the metadata (db.*) files should be accessed via a mount point that does not have the -llock NFS mount option; and
• The Perforce server journal file be kept on a local filesystem (i.e., not on a NetApp filesystem at all).

Please see the note in the Known Issues section for details, and watch this space for further updates.

Contents

Introduction         Background         Why Network-Attached Storage?         What About Performance?         What About Reliability? Network Appliance Filer Overview Perforce Server Storage         What is stored?                 The Perforce Metadata Database                 The Perforce File Archive                 Other Files         What Can (and Should) Be Stored on a Filer?         p4d with NFS         p4d with CIFS         Network Configuration         snap_checkpoint ing         NetApp Perforce Server Configuration         Filer Tuning         Advantages of Perforce-on-NetApp         Performance Measurements         Known Issues                 Server Type                 Other Issues Client storage Further Work Appendix: Filer Option Registry File Excerpts References Acknowledgments

Introduction

Background

This paper reflects my experience at Network Appliance, with Perforce running on Network Appliance storage, from September, 1997 through April, 2001.

In the summer of 1997, Network Appliance Engineering faced a period of rapid growth, both in the size of the department, and the number of concurrent development projects. It was apparent that our SCM platform (CVS, at that point), was among many systems that needed to be overhauled or replaced, in order to scale with our growth.

One of the many requirements for the new system was that it must allow us to use our own products as the underlying storage, both for the SCM repository and individual user workspaces.

Perforce caught our attention. Despite admonitions about the use of NFS storage in the Perforce documentation, our initial experiments, plus a few brief exchanges with Perforce support, convinced us that Perforce actually works fine storing its data on "NetApp filers" (as we commonly referred to the company, and its file server product).

Time bore out this conviction; we selected Perforce, and successfully deployed "Perforce-on-NetApp" over the next three years. Our Perforce server (and its storage requirements) scaled up with the rest of the engineering effort at NetApp. Since then, Network Appliance storage for Perforce installations has become a more common choice, particularly for large Perforce depots.

At Network Appliance, my roles included Perforce administrator, toolsmith, and trainer. I also maintained these roles with respect to our homegrown defect tracking system, and its integration with Perforce. I am not particularly an expert on NFS, networking, or on the internals of the Perforce server.

It should not come as a surprise if you detect a bias in favor of using Network Appliance storage products for storing Perforce data (in appropriate circumstances). For purposes of disclosure, I should add that, in addition to having worked for Network Appliance for several years, I remain a stockholder in Network Appliance. Also, the fact that I am now employed by Perforce Software should not be construed as any endorsement of Network Appliance products above any others. It has always been Perforce's aim to work well in all reasonable computing environments.

The goal of this paper is to sum up what we learned about running Perforce on NetApp storage, at Network Appliance. I hope that this will be useful for anyone else who is doing this now, or considering it.

Some of the information below pertains to Perforce with network-attached storage in general; some is specific to use with Network Appliance equipment.

Here's a spoiler: despite the length of this paper, there's not much special that you have to do in order to run "Perforce-on-NetApp", given that:

• You've selected a server host/OS with a sufficiently good NFS client implementation; and,

• You have a reasonable network configuration.

Perforce is oblivious to whether the filesystem is local or remote.

Why Network-Attached Storage?

The explosion of digital information in recent years has motivated the computer industry to invent new ways to store it all.

A decade ago, almost all online storage consisted of disk drives (sometimes configured into fancy RAID arrays), directly attached to general purpose computer systems, via various IO interfaces (SCSI, IDE, and so on). However, software and networking technologies for sharing access to filesystems between hosts via a network, including NFS, was fairly mature; Dedicating a host computer system for use as a file server was common, and special-purpose systems, designed from the ground up to serve files over networks, were beginning to emerge.

This parallels the shift that had occurred in the networking world, from general purpose computers dedicated to acting as routers, to the use of dedicated special purpose routers.

Today, several different technologies can be used to replace the familiar "direct-attached disk" arrangement, including architectures with the confusingly similar names "Storage Area Network" (SAN) and "Network-Attached Storage" (NAS).

NetApp filers are generally considered to be in the Network-Attached Storage category. (Though they can also employee SAN techniques). They are connected using familiar standard networking fabric and protocols, including Ethernet (copper or fibre), TCP/IP, NFS, CIFS, and so on. They serve access to files, whereas Storage Area Network technology is usually focused on serving access to disk blocks.

Some important benefits from this approach include:

• Storage/application host independence (multiple application servers and/or clients can directly access needed information. This yields a high degree of flexibility, without having to rearrange hardware)

• The "appliance approach" (the storage server can be streamlined and optimized to do one thing - serving data - very well)

• Leveraging existing expertise (since it uses standard technologies that users are already familiar with, along with their existing infrastructure)

What About Performance?

Until recently, suggestions to move large data sets for high performance applications (such as databases) from direct-attached storage, out onto the network, were often greeted with snickers or outright scorn. After all, basic considerations of raw bandwidth and protocol overheads had always kept direct-attached disks in the lead, performance-wise. SCSI is much faster than 10Mbps networks, and you wouldn't want to cripple your application server by making it access its data through a straw.

But over the past few years, gains in networking technologies - especially the advent of 1Gbps networking at reasonable costs - has substantially overcome these objections. As much as it goes against the intuitions we formed in the days of 10Mbps ethernet, accessing data though a filesystem over Gigabit ethernet can often outperform a locally-attached disk configuration. (In some part, this can sometimes be attributed to off-loading of running filesystem code from the server host).

For an example analysis, demonstrating that network-attached storage can yield performance equaling or surpassing host-attached disk, see:

<http://www.netapp.com/tech_library/3105.html> Performance of Competing Database Storage Technologies: NetApp Storage Networking vs. Veritas RAID¹

What About Reliability?

Another objection to using network-attached storage for critical applications is concerned with reliability: "A remote filesystem accessed across a network will necessarily be less reliable than a filesystem on a directly-attached device". This often boils down to doubts about whether acknowledgments of writes to a remote filesystem can be trusted, i.e., that the data has in fact "made it onto the disk". After all, many remote file systems rely on tricks about acknowledging writes before the data has actually been flushed to disk, in order to achieve good write performance. What happens if the power goes away, or the server crashes, and the data never makes it to disk?

These concerns are well addressed by modern implementations. Yes, writes may be acknowledged before the data has been flushed to disk, but not before the data has been captured in a non-volatile store. (Network Appliance filers journal writes into a battery-backed RAM, for example).

Another source of concern stems from uneasiness about the network in general: "Who knows what effect other network traffic might have on network-attached storage? After all, an IP network is more like a public highway, and a SCSI bus like a private expressway, right?"

Yes and no. Nobody would advocate connecting a Perforce server to its storage across the Internet, or a busy corporate LAN backbone. Network configurations are under our control, and there is nothing to prevent us from creating network segments dedicated to particular purposes, protected from random traffic.

Due to concerns like these, there has been a good deal of skepticism about the use of network-attached storage for important applications, such as:

Hah!  NFS has a long and invidious history of (ahem) being a tad
optimistic about what exactly has made it to disk just yet...

Do _not_ trust NFS with database I/O is a fine maxim to work by.

  - from a posting to the perforce-users mailing list, March, 1998

Network Appliance Filer Overview

Network Appliance file servers, are special purpose computers, designed to serve files via standard remote filesystem protocols - chiefly, NFS and CIFS (Microsoft's file sharing protocol for Windows).

A given filesystem can be accessed by NFS, CIFS, or both, simultaneously: hence, they are called "multiprotocol file servers".

From a hardware standpoint, filers are built from relatively plain, off-the-shelf board level components (though with a lot of attention to overall system reliability, and mechanical design for high maintainability).

Typically, a filer system consists of a "head" (CPU, memory, NVRAM, network and disk interfaces in one enclosure), and one or more "storage shelves", each housing some a number of disk drives. Current models allow up to 7 or 14 drives per shelf. Thus far, all filers have been single processor systems. The system is usually mounted in a rack.

A small-capacity (single storage shelf, top) Network Appliance F740 filer

As of this writing, the network interfaces are typically 100Mb/1Gb Ethernet, FDDI, and/or ATM; storage shelves are connected via Fibre Channel Arbitrated Loop. Individual disk drives can be up to 72Gb each, with total system capacities up to about 6Tb. (Minimal systems can be configured with a capacities of between 100 and 200Gb)

A dual-head clustered configuration is available, and can provide increased system availability (through a failover capability), and capacity (up to 12Tb).

Filer systems allow hot-plugging of storage shelves, disk drives, power supplies. All filesystems are RAID protected, so that the system can continue to operate in the event of a single drive failure within any RAID group. A "Snapshot" feature allows the state of any filesystem to be frozen in a matter of a few seconds, with a minimum of storage overhead. Various management functions (such as disk quotas, SNMP, NIS client support, and so on) are aimed at making filers easy to manage.

Filers run special-purpose software called Data ONTAP, an operating system and server application rolled into one. By foregoing the use of a standard, general purpose OS, filers can deliver a higher level of performance on a given hardware set than can a traditional OS. They are also easier to maintain; upgrading to a new major release of Data ONTAP usually takes only minutes, whereas upgrading to a new major release of, say Solaris, can take hours, or more.

Data ONTAP includes a patented technology called Write Anywhere File Layout (WAFL), that provides very good performance with RAID 4.

Perforce Server Storage

What is stored?

At a high level, a Perforce server comprises two main divisions of data: metadata and versioned files (present on all Perforce servers); plus a miscellaneous collection of supporting data, which varies somewhat from site to site, depending on local practice. Some of these (like the server journal file) are frequently accessed by a live server; others (like sets of saved server checkpoints) are not.

The Perforce Metadata Database

In part, Perforce is a specialized database system. All of the information beyond the actual contents of file revisions is stored in a relational database. This includes information such as the User Specifications, Client Specifications, lists of files synced to each client workspace, files open in each client workspace, revisions that have been integrated between files, and so forth. All of this information is stored in files named

db.tablename

in the "server root directory" (specified by $P4ROOT or the -r option when the p4d server process is started).

As a Perforce server performs a particular operation, it acquires locks on various of these files before reading or writing them. A locking order insures that consistency is maintained for table contents (and provides mutual exclusion) as the server services multiple user requests in parallel. Therefore, write locks on certain tables high in the locking order tend to lock out access to the entire server by all but the locker. Locks are placed at the file level; region locks are not used by Perforce servers to date.

Perforce relies on locking mechanisms provided by the host OS. Usually, these mechanisms work fine for locking files on NFS mounted file systems, especially where lock requests for a particular file all originate from the same host (in this case, the locking can be implemented entirely within the host OS, requiring no interaction with the NFS server).

A Perforce server, with all access to the database from a single host, should not, therefore, pose a great challenge to a network-attached storage configuration.

Some of the tables can grow quite large, be accessed very frequently. db.have is usually the largest - in the 10Gb+ range at some customer sites - and often very busy. Others will be smaller, and/or less frequently accessed.

However, you won't find the contents of your checked-in "hello.c#2" in any of the db.* files...

The Perforce File Archive

The second major division for data managed by the Perforce server is a set of one or more directory trees (corresponding to each local depot known by the server), rooted in $P4ROOT. Commonly (by default), there's just one: $P4ROOT/depot/ .

These trees hold all of the information needed to construct any revision of any file stored by the server. The revisions for any particular depot path are stored in one of a few formats, depending on the type the file; plain text files are typically stored in RCS file format (reverse delta-based); binaries as complete (compressed, by default) images of each revision, and so on. The format details are unimportant as far as direct-attached vs, network-attached storage is concerned.

The access patterns for these files vary from site to site, but correspond closely to Perforce activity for particular paths within the depot. As time goes by, there are likely to be large regions with relatively infrequent access (old branches that are no longer used), while other regions - perhaps a small subset of the entire depot, including "main" and active release branches - are accessed more frequently.

Other Files

The files described in the first two divisions (above) are bulk of the server's data, but there are other files created in the normal course of running a Perforce server. These can include:

• The server message log file (if enabled);
• The journal file (and previously saved journal files);
• Saved checkpoints;
• The server license file;
• Files used by custom local tools.

Some of these are present on almost all servers, and/or always reside in $P4ROOT (e.g., the license file). Others are optional, and may reside in arbitrary places. For example, while highly recommended, the creation of the journal and checkpoints is, technically, optional behavior, and it is usually recommended by Perforce that these files reside in a different filesystem than $P4ROOT does).

What Can (and Should) Be Stored on a Filer?

There's no reason why any of the server data described above cannot be stored on a filer, and in fact, at Network Appliance, all of it was - in a single filesystem. (Our confidence in the RAID protection inherent in NetApp storage allowed us to feel safe placing the journal in the same filesystem as the rest of the data).

(But see this note!)

This is contrary to some written Perforce documentation, i.e.:

In general (but in particular on Linux and FreeBSD) we recommend that the Perforce database, depot, and journal files be stored on disks local to the machine running the Perforce server process.

  - from the Perforce 2001.1 System Administrator's Guide²

Also, Perforce support has sometimes suggested to customers that network-attached storage be used only for the file archive trees, while keeping the database files on direct-attached storage.

Nevertheless, in our configurations, we never encountered performance problems attributed to the use of NFS storage for all of our Perforce server data.

We did encounter one problem, resulting in garbled entries in the journal file, in October of 1998. It turns out that writes to the journal file were not being done with locking. Perforce was relying on a Unix behavior wherein writes to a file on a local filesystem, opened with O_APPEND, are guaranteed to be atomic. But this is not true with writes to NFS mounted filesystems.

Perforce responded quickly, providing a patched p4d that fixed the problem by implementing locking on journal writes. Fears that this might cause performance problems on our (or other) platforms were dispelled, and this fix has been present in all p4d versions since Perforce release 99.1. (See the notes for change sets 8352 and 8186 in the Release Notes for Perforce³.

This was the only functional glitch we ever saw that was attributable to running Perforce with network-attached storage.

Of course, it is possible to configure a system such that any subset of the Perforce server data (down to the level of individual files) can be stored on a filer, with the remainder on a direct-attached device. Such arrangements might make sense in certain situations. But for the purposes of this paper (and reflecting our experiences with Perforce at NetApp), we'll assume that we want to keep all of the Perforce server data on a filer.

p4d with NFS

Network Appliance hosts its Perforce server process (p4d) on Unix servers, using NFS to mount the NetApp filesystem on the p4d host.

Initially, our Perforce server was hosted on a modest Sun Sparc/Solaris host.

Later (for infrastructure reasons that had nothing to do with Perforce), we moved to DEC (later, Compaq) Alpha/OSF1 hosts. (Details on the most recent configuration are shown below).

While we never experienced problems attributable to the NFS client implementations on these platforms, anecdotal evidence suggests that the NFS client code in some OS releases may not work as well (or reliably). In particular, lore has it that some versions of Linux and FreeBSD suffer from reliability problems when attempting to use NFS storage for a Perforce server. The reliability of file locking is the current favorite suspect; Further characterization of compatibility with these and other platforms remains to be done.

p4d with CIFS

Since we didn't run our Perforce servers under Windows at Network Appliance, I can't offer any substantial information about this configuration.

I did start a Windows Perforce server with storage on a filer via CIFS once, as an experiment, and noticed no ill effects after some very cursory testing.

Since CIFS is a stateful, connection-based protocol, users running servers with CIFS storage may need to pay special attention to any situation that can cause the loss of connections between the application server host and the filer. For example, with NFS, a file server reboot will usually be transparent to the NFS client (Perforce server), apart from a pause in service. With CIFS, a file server reboot will cause the loss of virtual circuit connections between the CIFS client (Perforce server) and the file server. This is likely to require some sort of recovery action on the CIFS client (Perforce server) host in order to resume normal operations.

It would be worthwhile for anyone interested in running in this configuration to do further testing. Be encouraged; Network Appliance has expended considerable effort toward making its CIFS support very compatible with Microsoft's. Many sites run large large Microsoft Exchange servers on NetApp via CIFS, and this would seem to bode well.

Network Configuration

For good performance, the single most important consideration when using Perforce-on-NetApp is in configuring the network(s) properly. Poor choices in network topology or configuration can result in unacceptable performance, while good ones result in performance comparable to (and sometimes better) than locally attached disks.

There is no single prescription that will fit all sites; the choices will be influenced by the size and patterns of the particular workload, but some general guidelines always apply.

First, think about the network traffic that network-attached storage adds to the picture. When running a Perforce server with network-attached storage, the overall network bandwidth demands on the host will increase roughly by the amount of data transfer that would, otherwise, flow across the hosts disk interfaces. For a busy Perforce server, this can be a substantial amount of data!

To accommodate this bandwidth demand, and to isolate it from the separate bandwidth requirements for Perforce client connections to the Perforce server, it will usually be important to choose a fast network, and a dedicated interface to attach the filer to the host.

At NetApp, we initially used 100Mbs, and later, 1Gbps Ethernet.

This dedicated interface can in fact be a private back-to-back connection, or an equivalent connection through a network switch. It may help to think of a back-to-back connection between a server host and a filer as, simply, a replacement for an IO Bus (e.g., SCSI) connection to a disk array.

Which approach to use can be influenced by considerations including the existing infrastructure, budget, security requirements, and so on. The important thing is to insure a guaranteed pipe, of sufficient size, between the Perforce server and its data.

The server host will then also need a separate interface (or interfaces) to accommodate connections from Perforce clients.

Except for modestly busy sites, it can be a big mistake to share a single network interface for both Perforce client access and access to the filer; the bandwidth demands for each will tend to peak at the same time. Consider the initial p4 sync of a large client workspace: the server will be busy writing records to db.have, reading the file archive to retrieve the revisions of files being synced, and feeding the synced revisions out to the client, all at once.

At Network Appliance, the current configuration for servers (including compute servers, build servers, and the Perforce server) is generally

• servers are cross-connected to the filers they need via a 1Gbps Ethernet switched "back-end" fabric, and:

• servers are connected to other servers, and out to desktops, through a separate "front-end" fabric (Also 1Gbps at the host interfaces).

The following diagram illustrates the general approach:

For further information on the nuts and bolts of configuring networks properly for use with NetApp filers, see

<http://www.netapp.com/tech_library/3112.html> Architecting a Gigabit SAN for Database Application Environments ⁴

snap_checkpointing

As our Perforce depot grew, we noticed our daily Perforce server checkpoints taking longer to complete. During a checkpoint, the Perforce server locks the entire database, effectively taking the server offline. In the beginning, out checkpoints were taking only a few tens of minutes, but over time, as the database grew, they grew to taking over an hour.

Of course, we ran checkpoints in the wee hours, but of course, the sun never sets on software developers - especially as you have more of them, and more geographically distributed.

Fortunately, we were able to solve this through the use of NetApp filer Snapshots.

Basically, with this approach, the Perforce server is locked, then a command is issued to the filer to take a snapshot; the lock is released, and the Perforce server can resume the servicing of user requests. This whole process completes in just a few seconds.

Next, a p4d -jd command is issued, with $P4ROOT set to point at the $P4ROOT directory in the NetApp snapshot. This runs to completion in the background, taking however long it takes.

If you're interested in doing this, please refer to

Fast-Checkpointing a Perforce Database Using a Network Appliance Filer⁵

Note: snap_checkpoint users should be aware of an incompatibility between Data ONTAP releases 6.12 (or later), and all existing versions of p4d, up to 2001.1 (at least). This should be fixed in a future release of p4d. Please refer to the document referenced above for further details.

NetApp Perforce Server Configuration

As of September 5, 2001, the server configuration in use at NetApp was:

Server Host

COMPAQ AlphaServer DS20E 500 MHz (1 cpu)
4 Gb memory
Dual Gb ethernet interfaces
OSF1 V4.0 1530

Storage: Network Appliance F760 filer

1 Alpha CPU
1 Gb memory
42 9Gb disks totaling 378Gb (raw)
Dual Gb ethernet interfaces
Two filesystems (1 for Perforce, one for defect tracking)
170Gb filesystem dedicated to Perforce
NetApp Release 6.1.1R1
NFS V3 UDP mounts

Perforce Server

Server version: P4D/OSF/2000.2/21189 (2001/03/14)
Server license: Network Appliance 345 users
Number of files in the depot: 877241
Number of client specs: 7088
Number of change lists: 127230
Number of job records: 52418
Metadata size: db.have: 9.2Gb; db.*: 10.2Gb
Number Data ONTAP/Netcache source branches: 131
Files in the main branch: 10267

Filer Tuning

We never needed to do any special tuning of Data ONTAP options on the filers we used for storing Perforce data.

Contrary to oft-repeated lore, there is generally no reason rooted in performance or reliability for running NFS over TCP (rather than UDP), in LAN configurations. (TCP may be preferable when using NFS across WANs).

For reference, the relevent filer configuration options we used at NetApp are reproduced in Appendix: Filer Option Registry File Excerpts.

Advantages of Perforce-on-NetApp

So, why would a Perforce site want to run with Network Appliance storage, anyway?

• This is probably of interest to larger sites, with large databases. Perforce runs great on small hardware sets, and the size of Perforce sites varies a lot. Many Perforce servers have only a few users, but many have hundreds, and some even thousands. Also, big sites tend to be less tolerant of Perforce downtime (whether scheduled: e.g., daily checkpoints, or not, e.g., disk failures).

• NetApp filers allow filesystems to be expanded dynamically, with no interruption of service; when you run out of space, you can simply plug in another disk, tell the filer to add it to your filesystem, and presto: more space.

• Filers have baked-in RAID data protection. When a drive in a RAID group fails, the filesystem keeps on serving data, at a degraded rate. The bad drive can be replaced (using either a hot-spare, or by plugging in a replacement). The data is then reconstructed onto the new drive, at which point normal performance is regained.

• The Snapshot feature allows for "instant" Perforce checkpointing. This can be important for large Perforce servers, where the time required to write a checkpoint (during which the Perforce server is unavailable) can be large.

• The filer's "SnapRestore" feature (instantly revert a live filesystem to a previously snapshotted state) can be valuable in setting up testing baselines. This may be useful if you need to qualify new releases of Perforce with local tools, for example. For further information on this approach, see

  <http://www.netapp.com/tech_library/3055.html> Creating a UNIX-based Database Software Testing Environment Using a NetApp Filer⁶

• Filers support a wide range of management tools to make it easier to monitor and manage the state of the filesystem and large data sets.

Performance Measurements

Unfortunately, there are as yet few well-defined ways to measure and compare the performance of Perforce in various configurations.

Questions about the viability of running Perforce-on-NetApp came to my attention from time to time, both from the Perforce user community at large, and from prospective NetApp customers specifically interested in doing this. The leading concern is usually performance.

Unfortunately, while we had been running Perforce-on-NetApp in production for years to good effect, we had no real performance metrics to refer to.

I was, at least, able to rig a rudimentary facility for measuring a well-defined Perforce workload in different server host/storage configurations.

For each test, against a newly-initialized Perforce server, five phases were timed, using a gcc source code distribution as test data (about 2800 files):

initial_add		time to p4 add the files in a new client workspace
submit		time to p4 submit the newly added files
new_sync		time to p4 sync the files into a second new client workspace
flush_to_#none		time to do a p4 flush ...#none in the original workspace
flush_to_head		time to do a p4 flush ... in the original workspace

These tests were run repeatedly, both with filer-based (NFS) and local disk storage.

To summarize: the "initial_add", "submit" and "new sync" operations always completed significantly faster when using NetApp storage; and the "flush_" operations always completed in comparable times to the local disk tests. (Sometimes somewhat faster, and sometimes somewhat slower).

These results were posted to the perforce-users mailing list in February, 2001:

[p4] network appliance vs local disk?⁷

The scripts involved are available from the Perforce Public Depot:

public.perforce.com:8080//guest/richard_geiger/p4bench/⁸

These tests were admittedly simple, but certainly tended to bear out the idea that the performance of network-attached storage can, at least rival that of host-attached disks.

Known Issues

Server Type

Prospective users should bear in mind that all NFS client implementations are not created equal, and lore has it that some hosts/OS combinations do better than others at running Perforce with NFS mounted data.

Personal experience leads me to:

• recommend running p4d on Sparc/Solaris or Alpha/OSF1 hosts. I believe that these choices work well. (And that there are probably other good choices that I just haven't used).

  Note: Under Solaris (by default), requests made by the Perforce server to lock metadata (db.*) and journal files result in "network locking" being done between the Perforce server and the filer. This can cause noticeable performance effects for operations that generate lots of locking requests. This effect is not readily noticeable for most Perforce operations, but may be involved in some of the performance issues noted in "Other Issues", below. Generally, such performance effects are most pronounced for operations that do large updates to the metadata, as this results in large numbers of writes to the journal, which is is locked and unlocked on a per-record basis. Solaris supports a scantily-documented NFS mount option, "-llock" ("local locking"), which disables the network locking requests. (See the nfsstat(1M) man page and /usr/include/nfs/mount.h). An earlier revision of this paper recommended the use of the -llock mount option for Solaris Perforce servers with data on NetApp filers. Recent feedback from sites running Solaris Perforce servers on NetApp storage has cast doubt upon the reliability of this configuration. I now recommend, for optimal performance and reliability on Solaris hosts with NetApp storage for the Perforce server that: The -llock should not be used with Perforce, until further testing can be performed. Specifically, the metadata (db.*) files should be accessed via a mount point that does not have the -llock NFS mount option; and The Perforce server journal file be kept on a local filesystem (i.e., not on a NetApp filesystem at all). Tests indicate that OSF1 V4.0 hosts do not do network locking for Perforce on NFS. Users considering other operating systems should be aware of this, and investigate the implementation of flock() on their host. (As of r2001.1, the Perforce server uses flock() locking for all Unix hosts except for Irix and QNX (which use fcntl()).

• caution prospective users of NetApp filers and Linux or FreeBSD Perforce server hosts, with which some people have reported problems in the past. (However, these system continue to evolve, and newer releases may solve these problems)

• encourage users to experiment with other hosts of interest.

Other Issues

• One site has reported significantly slower Perforce database rebuilds (p4d -jr), as compared with the same operation with locally attached storage. (about 10x slower with pre-2001 p4ds; 2x with 2001.1). The cause is not yet well understood. This shouldn't be of much concern, unless for some reason your site performs database rebuilds frequently.

  The same site reports dramatically slower (as compared to local disk) "p4 changes -i path" commands, when using a pre-2001.1 p4d. However, they report that the same commands complete dramatically faster for the same commands in all configurations (local- or network-attached storage) with 2001.1 Perforce servers.

  ...But that performance is otherwise comparable to locally attached disk in their tests.

• Network Appliance has not yet upgraded to Perforce 2001.1. As far as I am aware, this is not due to any known problem with 2001.1 (On the contrary, 2001.1 seems to bring significant performance gains in general).

• At NetApp, and at another large Perforce-on-NetApp site, we independently observed "server swamp" caused by frequent deletion (p4 client -d) of large (10-30K file) client workspaces. Further analysis indicated that the Perforce server locks the database for the duration of this operation, as records in the db.have table are deleted. This causes commands from other Perforce clients (even short-running ones) to block, waiting for the "client -d" to finish.

  It's not clear as to whether these effects have anything to do with running Perforce-on-Netapp or not. It may very well be that this is simply a matter of scale; creating and tearing down 10,000-file client workspaces every few minutes makes for a busy Perforce server in any case!

  At Network Appliance, these problems were solved by adjusting our process to make more efficient use (and re-use) of large client workspaces.

  With advice from Perforce support, we also found it helpful, when deleting large client workspaces, to cause the deletion of the client's db.have records using a p4 sync or p4 flush before issuing the "p4 client -d", for example:

  p4 flush //clientname/...#none"
  p4 client -d clientname
  rm -rf clientdir
  

  This helps because "flush" and "sync" do db.have record deletion in "batches"; the database files are locked and unlocked multiple times during the deletion of a large client workspace, affording other p4d processes a chance to run.

• Of course, having your data on a network filesystem opens the possibility that the data can potentially be accessed by people or programs from unauthorized hosts. Set up access controls on such filesystems accordingly.

• Be aware that NFS write data is being cached, and that it is theoretically possible that data written by the server (and acknowledged by the file server) will not make it to disk. However, in my experience, with Network Appliance filers, this has never happened to me. It's an extremely rare occurrence, typically requiring multiple simultaneous hardware and maintenance failures.

  In the unlikely event that it ever does happen to a Perforce server, the data lost would be a small amount of the most recent Perforce activity. Recovery would be a simple matter of rolling back to a known good state just prior to the failure, using standard perforce database recovery techniques.

Client storage

Perforce client workspaces stored on NetApp filesystems work fine. There are no special considerations. At Network Appliance, most workspaces were in fact kept on NetApp filesystems, both Unix (via NFS) and Windows (via CIFS) Perforce clients.

This affords some advantages:

• Keeping a client workspace on a shared filesystem makes it easy to do development and builds for multiple host architectures in a single workspace (provided the build machinery is set up to support it).

  This can also work for both Windows and Unix hosts in the same workspace, since filers allow the same files to be shared via both NFS and CIFS.

  Until recently (Perforce 2001.1), this required extra care because of different end-of-line conventions in Unix and Windows; the addition of the "LineEnd: share" client option in Perforce 2001.1 makes it easier.

• The filer's "Snapshot" feature is always nice to have. For users, it affords the ability to retrieve recent versions of files after accidental deletion, without having to ask systems administrators for a restore.

Further Work

Our experience in running Perforce-on-NetApp at Network Appliance has convinced me that this is a viable solution with real benefits, especially for large Perforce sites.

Further exploration of these techniques, on the part of Perforce Software, Network Appliance, and/or intrepid Perforce administrators, is justified.

Further testing (and testing tools, both for reliability and performance) would be welcome. These would be helpful both to better understand the impact of network-attached storage, and other configurations.

It is bothersome (wearing my "Open Source Engineer" hat) that Linux and FreeBSD are reputed to have problems running p4d with NFS storage. I'd like to investigate further, and tell you more about whether indeed there are problems, what they are, and how they can be fixed.

It would be interesting to explore even more involved configurations, aimed at better capacity and performance for very large Perforce depots. For example, is there a point at which it makes sense to divide the server data up between the data and the file archive, and dedicate network interfaces, filesystems, or even filers to each?

Appendix: Filer Option Registry File Excerpts

The following are the options pertaining to NFS, from the filer used to store Perforce server data at Network Appliance, as of September 5, 2001.

#!Registry V1.0 begin: Sun Sep  2 10:18:12 GMT 2001
default.options.if.default.flowcontrol=send
default.options.if.default.mediatype=auto
default.options.if.default.trusted=on
default.options.if.default.wins=on
default.options.nfs.udp.xfersize=32768
options.cksum_offload.gbeII=off
options.cksum_offload.tcpudp.recv=on
options.cksum_offload.tcpudp.xmit=on
options.disk.checksum_blocks.enable=on
options.disk.read_after_write_verify.enable=off
options.disk.shm.enable=on
options.if.e0.addr=`hostname`-e0
options.if.e0.mediatype=100tx-fd
options.if.e0.netmask=255.255.255.0
options.if.e7.addr=10.nn.nn.nn
options.if.e7.netmask=255.255.255.0
options.if.e8.addr=10.nn.nn.nn
options.if.e8.netmask=255.255.255.0
options.if.e8.wins=off
options.ip.fastpath.enable=on
options.ip.match_any_ifaddr=on
options.ip.path_mtu_discovery.enable=on
options.ip.sack.enable=off
options.nfs.big_endianize_fileid=off
options.nfs.mount_rootonly=on
options.nfs.per_client_stats.enable=off
options.nfs.require_valid_mapped_uid=off
options.nfs.tcp.enable=on
options.nfs.tcp.recvwindowsize=26280
options.nfs.tcp.xfersize=32768
options.nfs.udp.xfersize=32768
options.nfs.v2.df_2gb_lim=off
options.nfs.v3.async_writes=off
options.nfs.v3.enable=on
options.nfs.webnfs.enable=off
options.nfs.webnfs.rootdir=XXX
options.nfs.webnfs.rootdir.set=off
options.raid.debug_parity_inconsistencies=on
options.raid.verify_blockmap_bufs=on
options.sysconfig.periodic_check=on
options.sysconfig.periodic_errors=syslog
options.sysconfig.periodic_interval=60
options.thresholds.cpuBusy=90
options.thresholds.fsFull=98
options.thresholds.fsNearlyFull=95
options.udp_lg_dgram.xmit_cksum.offload=off
options.wafl.async_resize.enable=on
options.wafl.convert_ucode=off
options.wafl.create_ucode=off
options.wafl.default_nt_user=
options.wafl.default_security_style=unix
options.wafl.default_unix_user=nobody
options.wafl.log_oplocked_writes=on
options.wafl.maxdirsize=10240
options.wafl.nt_admin_priv_map_to_root=off
options.wafl.root_only_chown=on
options.wafl.wcc_max_entries=10000
options.wafl.wcc_minutes_valid=20
#!Registry V1.0 end:

References

1. Morgan, Dan & Browning, Jeff, Performance of Competing Database Storage Technologies: (http://www.netapp.com/tech_library/3105.html) NetApp Storage Networking vs. Veritas RAID, Network Appliance, Inc.

2. Perforce 2001.1 System Administrator's Guide, Perforce Software, Inc., July 2001.

3. Release Notes for Perforce, Perforce Software, Inc., July 2001.

4. Lenzer, Michael, (http://www.netapp.com/tech_library/3112.html) Architecting a Gigabit SAN for Database Application Environments, Network Appliance, Inc., 2001

5. Geiger, Richard, Fast-Checkpointing a Perforce Database Using a Network Appliance Filer, Network Appliance, Inc. & Perforce Software, Inc, January, 2000, revised September 2001.

6. Browning, Jeff, (http://www.netapp.com/tech_library/3055.html) Creating a UNIX-based Database Software Testing Environment Using a NetApp Filer, Network Appliance, Inc.

7. Geiger, Richard, [p4] network appliance vs local disk?, perforce-user mailing list, February 2001.

8. Geiger, Richard, "p4bench" performance measurement scripts, The Perforce Public Depot, August, 2001.

Acknowledgments

Thanks to Network Appliance, Inc., for providing access to information and permission to reproduce some of the illustrations used herein.

Several people at Perforce (Notably Gerry Thompson and Shelley Reed) deserve thanks for giving valuable feedback and reviews.

Thanks also to all of the people in support@perforce.com, and all those who back them up (past and present). I hope that I can help to uphold the standard of excellence they showed me as a Perforce customer.

p4://chinacat:8080: $Id: //depot/user/rmg/p4ntap/User_Conference_2001_rmg.html#3
$DateTime: 2001/10/23 16:01:48