How to Plan an Easy Server Upgrade
Many of our customers have a large number of server instances (including different types of replicas), with thousands of users working with a variety of client applications (both from Perforce and third-parties). This level of complexity can be concerning when upgrading to a new server version.
But routinely updating your Helix Core server delivers benefits. Being on a recent version is a key requirement for support. Plus each Helix Core release has numerous small enhancements and addresses issues. And depending on what version you are on, you could be missing a lot of great new features. Most importantly, Perforce is constantly improving performance. Developer productivity and high-performance automation are the keys to getting the most out of your investment in Helix Core.
Most Helix Core installations are mission-critical for our customers, and they involve a lot of moving parts with integrations, tooling, and environment settings. So, for any installation beyond a single server, contemplating a server upgrade requires planning. With the right plan, surprises can be all but eliminated.
We are going to focus on tips for large enterprise customers who have more complex environments. For smaller teams, you may not need to follow all these tips, but you will likely benefit from doing so.
Why Wait To Upgrade Servers?
The actual software metadata version upgrade for Helix Core takes, on average, less than 30 seconds to complete on a single server. Steps include:
- Stop the service.
- Swap binaries for new version.
- Execute a command similar to p4d -r $P4ROOT -J $P4JOURNAL -xu (with appropriate values for P4ROOT and P4JOURNAL).
- Restart the service.
For customers with multiple servers, updating your entire infrastructure can be quick. Many customers use tools like Ansible, Puppet, and Chef to speed deployments.
Pushing out the server upgrades is the easy part. It’s ensuring everything else still works that can make things complicated. But with preparation, it’s possible to avoid issues. To successfully upgrade, we recommend auditing, testing, and configuring your system with an eye toward present and future needs.
Step 1: Map Your Topology
If you haven’t already, take inventory of your entire development environment. Start by recording configs and versions of everything, by location, in a spreadsheet. Check your licenses. What are the clients for users and automation, where are they, and what are they doing? Which ones are the most important for your organization?
You want to look at where activity is occurring. Check who or what is interacting and against what servers, replicas, edge servers, or proxies.
As an example, the following command on Linux analyzes standard server logs for different client versions:
grep -a -A1 "Perforce server info:" log | grep -a -vE "Perforce server info:|^\-\-$|\t\S+ \S+ pid [0-9]+ (compute end|completed)" | perl -ne 'print "$1\n" if /^[^\+\[(.+)\] /' | sort |uniq -c | sort -k 1,1 -n
218 p4/2016.2/LINUX26X86_64/1468155 276 CommitTrigger/v81 376 P4V/NTX64/2017.3/1590419/v83 481 unknown/1.50.6550.21600 1681 P4VS/2017.2.159.0319 1743 Builder/22.214.171.124 1789 SWARM/2017.1/1517929 2242 Pipeline.Automation/126.96.36.199
We can see some standard Helix client programs and custom tools with version strings such as “Builder” and “Pipeline.Automation” have been developed in the organization. It’s easy to find out IP addresses and, thus, the machines from which they are being run.
Tracking server interactions can help you determine what needs to be tested. It helps ensure you’re not missing any components. And you can use this information to map your environment.
This takes some time and planning, but it’s a vital step for things going smoothly. Once you know how things relate, you can start making decisions about how to upgrade.
Obviously, if you are monitoring things, examining logs, and/or have dashboards, you already know much of what is going on. Share this information with your infrastructure team. It is extremely useful for troubleshooting issues during operations and essential to interacting with Perforce support, too.
Step 2: Test Clients and Integrations With Helix Core
The majority of upgrade challenges tend to occur with custom client programs and other systems interacting with the Helix Core server. Issues can arise, for example, when a custom client, trigger, or tool runs a command and doesn’t get back what the data it is expecting.
For example, there may be a new field in the output of a command due to server enhancements. Sometimes, even a simple thing like a version mismatch between a server and an API could cause automation to break. Occasionally, an old installed p4 client on a slightly neglected system can be many releases behind the installed server.
The more complicated the environment, the greater the risk for client issues. And although this type of issue is rare and usually easy to fix, it is easily avoided.
Having a test instance of your server allows you to check that integrations are compatible between upgrades. Custom built integrations that were installed to match the server may need to be updated along with Helix Core.
To perform a test:
- Copy a production checkpoint and restore to a test instance.
- Copy across as much of the depot/archive files as you need for testing the upgrade (typically a small subset of the full repository).
- Test the various tools and integrations, including custom configurations. This should include triggers and other workflow enhancements.
- If required, fix issues in the client programs before proceeding with the upgrade.
Testing the new version lets you discover problems on a smaller scale. So when you’re upgrading, you’ll avoid surprises, which means minimal downtime for your teams.
Step 3: Automate Server Upgrades Using the Server Deployment Package (SDP)
Once you have mapped the environment and tested clients and integrations, you should be ready to proceed with your upgrade.
But let us give you one more thing to study. To help enterprises of all sizes efficiently upgrade and minimize server downtime, the open-source Server Deployment Package (SDP) is available from Perforce. SDP includes a standard set of scripts that admins use, and it can help provide consistency of management across all of your instances and servers.
Once installed and configured, it also allows enterprises to easily upgrade. This automates upgrades for multiple servers. The traditional advice was to start on the outside edges of your environment and work inward (replicas before master). But we now recommend upgrading everything at the same time.
How Does SDP Work?
When using the SDP Package for an upgrade, you only need to complete two steps for an instance on a machine:
- Download the new version of P4 and P4D.
- Run upgrade.sh <instance>, e.g.
Behind the scenes, SDP does the work. It stops the server, provides a clean “upgrade point” via a journal rotation, updates the database on the live and offline directories, and restarts the server.
But upgrading is just one aspect of SDP.
Planning Your Future Using Perforce
Some customers are cutting-edge adopters and are always on the latest release. Other customers that too busy, and they only update every few years. For both of these enterprises, SDP can be an integral part of your solution.
SDP provides proven basic management and best practice features, including init scripts and checkpoint scripts. The package includes:
- Resources to plan, deploy, and manage sophisticated topologies.
- Standards for developing custom triggers.
- Best practice configurables and other product settings.
- Shell environment management standards.
- Supplemental maintenance scripts for admins – including cleanup tasks, removing old/unused workspaces, and deleting labels.
Proper installation can be a differentiator for live product and eval/PoC deployments at any scale. SDP is mature and sophisticated enough to meet complex needs of many demanding environments with no customization. And it delivers a framework for extension that helps contain the complexity of custom deployments.
Improve Your Environment; Make Your Server Upgrades Easy
Upgrading more often, while requiring the allocation of time perhaps twice a year for major upgrades, will mean that the magnitude of changes and variables will be less with each instance. Plus, keeping on top of current server usage (e.g., the variety of custom clients in use) and running a standard test installation, makes upgrading a low-risk activity.
And as technologies change, so does the SDP. We encourage our Perforce community to communicate and make changes. It is continuously receiving and incorporating input from Perforce Consulting, Support, and Engineering.
Now you’re ready to start making a plan to upgrade. Check out the latest releases of Helix Core. Features include:
- Leverage WAN acceleration technologies to better support remote teams.
- Boost your parallel sync performance with improved sever resilience under load.
- Improved failover. One command replaced multiple triggers.
- Tightened security (including multi-factor authentication and SAML 2.0 authentication).