January 12, 2011

Three Level Digital Asset Classification

What's New
E-Commerce
Traceability

Many sites in the defense and government communities will be familiar with the need to maintain several levels of digital asset classification.  For the sake of simplicity, let's say that we need to maintain source code at these levels:

  • Green: open to all developers, constitutes the majority of the source base.
  • Yellow: an intermediate classification, not available to all developers.
  • Red: the highest level classification, available to a small group of developers.

To make things interesting, let's assume that Yellow code can't exist on a Green server, and Red code can't exist on a Yellow or Green server.  In other words, because of stringent security requirements, Yellow and Red source code is physically isolated from lower level systems.

Let's further assume that development for the Green code is done exclusively on the Green network, the Yellow code is developed exclusively on the Yellow network, and the Red code is developed exclusively on the Red network.

When we run and test the code on the Green network, we have only Green code, and so we have a partially functional system.  (The system may be able to operate, but will not have full capabilities.)

When we run and test on the Yellow network, we have Green and Yellow code.  The system is more capable here, but still not full featured.  In order to run and test on the Yellow network, we import the latest version of the Green code before building and compiling.  If we need to make a change on the Green code, we may do a patch, but we have to officially commit the patch back on the Green network.

Finally, when we run and test on the Red network, we bring in the latest versions of the Green and Yellow code, and we get a fully functional system.  Any problems discovered in the Green and Yellow code may be patched, but cannot be officially committed here.

As a final complication, we assume that the Green, Yellow, and Red servers cannot communicate with each other.  The networks are completely isolated.  This complication is really the fly in the ointment -- if the networks could communicate with each other, we could use a standard remote depot solution.

Depot Structure

The question then becomes: How can we maintain our three code bases effectively in Perforce?  We want to easily manage the problem of periodically moving code from one system to another, and avoid an unnecessarily complicated build and release process.

At the Green level, things are very simple.  Let's assume we have one product called Ace.  We'll have a depot called Green.  Starting out, we just have three files:

> p4 files //Green/...

//Green/Ace/src/green.c#1 - add change 636 (text)

//Green/Ace/src/red.c#1 - add change 636 (text)

//Green/Ace/src/yellow.c#1 - add change 636 (text)

The file green.c is the only real source file.  The other two files are placeholders at this level; they are present to allow our build system to work correctly.  (Rather than placeholders, they may be stubs that provide some simple functionality for testing.)

At the Yellow level, we provide a second depot called Yellow.  This depot holds only the files that must be secured at the Yellow level.  Having a separate depot provides a few advantages:

  • A nice organizational element, which we'll examine in the next section;
  • the ability to physically put files at the Yellow level on different physical storage;
  • and an easy way to manage different permissions for files at the two levels.  For instance, we can easily grant write access to the Yellow depot, and only open access to the Green depot, to make sure that no changes to Green code are submitted here.

For our example, let's say that we only have one file maintained at the Yellow level:

> p4 files //Yellow/...

//Yellow/Ace/src/yellow.c#1 - add change 637 (text)

Finally, at the Red level, we have a third depot, called Red.  Again, only the files that must be maintained at the Red level are managed in this depot.

> p4 files //Red/...

//Red/Ace/src/red.c#1 - add change 638 (text)

So how do we move code from one server to another?  For the sake of simplicity, we can just do code drops.  Let's say we reach a good development milestone on our Green code.  We can take a snapshot of the Green depot, move it to the //Green depot on the Yellow server, and import it there.  That can be done using P4V's Reconcile Offline Work tool or a similar script.  We can test on the Yellow server, and finally move snapshots of the Yellow and Green depots up to the Red server.

This approach of course does not preserve full history of the Green depot on the Yellow or Red servers, but it does give us important baselines or snapshots of the Green code to use for testing and development of the Yellow and Red code.  Since all official development of the Green code has to happen on the Green server, this trade-off is acceptable in most cases.

Now let's see how we can use this depot structure effectively.

Workspace Management

At the Green level, our workspaces and build systems are simple.  At the Yellow level, we use an overlay mapping in our workspace to superimpose the Yellow code:

View:

//Green/... //local.yellow/Green/...

+//Yellow/... //local.yellow/Green/...

Using this mapping, our real Yellow source code file is put into our workspace, instead of the placeholder/stub version.  Our build system does not need to be changed.

At the Red level, we add one more mapping to overlay the Red depot:

View:

//Green/... //local.red/Green/...

+//Yellow/... //local.red/Green/...

+//Red/... //local.red/Green/...

This view gives us the real Red source code file, rather than the placeholder/stub.

As discussed in the previous section, we can easily set permissions on the three depots to make sure that only files at the server's own security level are modified.  Files at the other level(s) may be opened but not submitted.

If necessary, we can use triggers to enforce these conventions in workspaces at the Yellow and Red levels.

One thing to watch out for is (surprise!) renamed files.  If a file is renamed at the Yellow or Red levels, then its placeholder/stub at the Green level must also be renamed.