Game development and virtual production studios use game engines to create the unbelievable. But with game engines— like Unreal Engine or Unity — project assets are stored as large binaries. To efficiently collaborate on and manage these assets, version control systems need to treat them differently than other file types.
Most version control companies cannot manage these binary files. But Perforce P4 (formerly Helix Core) — the version control standard for game and media — is known for being able to manage binary assets and with the speed teams need to move fast. It is important to set up P4 to handle binaries correctly. Read this blog to learn how to manage binary assets by creating a Perforce typemap.
Feel free to read along or jump to the section that interests you the most.
Table of Contents
- What Is a Perforce Typemap (P4 Typemap)?
- What to Know — Perforce Typemap
- Upgrading to UE5?
- Git + Different File Types
- Don’t Ignore P4 Ignore
- What to Do When Something Goes Wrong
- Not a P4 Customer?
➡️ Version Everything — Without Limits
What Is a Perforce Typemap (P4 Typemap)?
A Perforce typemap tells P4 exactly how you want to handle files with specific file extensions or within certain folders. Although P4 will attempt to detect file types and store them correctly, a Perforce typemap removes any uncertainty, especially when dealing with modifiers (more on that later).
Looking for Set Up Instructions?
If you are new P4 customer, start setting up your depot and stream.
What to Know — Perforce Typemap
If you are using P4 and a game engine, it is critical to your project to set up a typemap BEFORE you start working on files. This is because the typemap will only impact files that are submitted after the typemap created. Once created, your typemap will identify which files to store as binary or text, enable file locking, and more.
Get the Complete Game Engine Overview >>
File Types
There are several specialty filetypes that can be stored in P4. But there are two main types — text and binary.
Note: For international users, review file types for utf-8 and utf-16 information.
| File Type | Description |
|---|---|
text | Files are stored as text, with revisions as reverse deltas. Line endings are automatically translated between operating systems. |
binary | Files are stored as binary, with revisions stored as complete files. Binary files are compressed on the server by default to save space. |
File Type Modifiers
Modifiers are what really matters when it comes to your typemap. They allow you to designate different functionality. For example, one of the most important features in P4 is exclusive locking. This allows a user to check out a file and prevent other people from overriding their work. It is best practice when working on game engine files to always check it out and lock it. But this feature is not enabled as a default. The modifier is what makes this happen.
We will list a few types of modifiers here, but you can find a more complete list in the documentation. Note: After the filetype you can add flags with a + character and then the flags.
| Flag | Description |
|---|---|
+w | Set file to always be writable in workspaces. For example, certain log files and build files. |
+F | Store the file uncompressed on the server. By default, binary files are compressed on the server. This is useful for certain files that are already compressed — such as JPGs and GIFs. The server won't waste time compressing them again. |
+l | Set exclusive locking for a file. When this file is checked out, the file will automatically be locked for other users so they cannot edit. This is useful for binary files that should not be edited by multiple users at the same time. For example, certain game engine files, 3D models, movies, etc. |
+S<n> | Only store the most recent versions of this file. The <n> needs to be replaced by the number of revisions to keep. For example, +S2 will store the most recent two versions.Useful for especially large files if you don't need all their versions or for files that change often but you don't want to store all of the changes. This can help you save space on your server. |
These flags are placed immediately after the filetype and can be combined with each other. For example, binary+FlS4 will store the file as binary, full file (uncompressed), with exclusive locking, and only store the four most recent versions of the file.
You can also specify a flag without a filetype in your typemap. For example, +w //.../writeable/... will set all files in the writeable folder to be writable.
Pattern
The pattern is the depot path of the files that the typemap matches. Patterns start with // and then contain a path including the depot name, folders, wildcards, and file extensions.
* is a wildcard that matches any number of characters, except slashes.... is a wildcard that matches any characters through all subdirectories.
Pattern Examples
For example, set all .obj files inside any folder called “models” to be treated as binary, stored uncompressed, and use exclusive locking. This is a good way to treat 3d models stored in obj format.
binary+Fl //.../models/*.obj
Set all .obj files anywhere underneath a folder called “bin” to be treated as binary, always writeable, and only store the latest 3 revisions. When compiling C++ code, linking files are stored in a build directory, also with the .obj extension. They need to stay writeable because users don’t directly check them out.
binary+wS3 //.../bin/....obj
Upgrading to UE5?
Learn how to seamlessly manage an Unreal Engine integration with Perforce Streams. Upgrade without slowing down.
Git + Different File Types
If you are coming from the world of Git, typemaps are probably a new concept. This is because in Git, you are just storing source code. Some teams do use Git LFS to store large files. You can identify file extensions you want stored separate from Git and then use code to point to the location of the binary asset.
For many teams, this is extremely hard to manage at scale. Even if you only have one project, you must do this for every file and every version. This severely complicates build pipelines, which is why so many teams choose Perforce.
Find out more about Git vs. Perforce >>
Don’t Ignore P4 Ignore
Another important part of setting up your typemap is to identify what files need to be ignored from version control. These files will not be locked and will be excluded from P4. They will not be picked up when running Reconcile Offline Work... or p4 reconcile
We recommend that all projects have a .p4ignore file in the root of the project. This should be used to avoid submitting and syncing unnecessary large files (like intermediate build files or baking data), temporary files that change frequently (like log files), and user-specific settings and configuration files. This will let you sync faster, avoid confusion, and avoid accidentally sharing private user data.
Just like with the typemap, p4ignore will only impact files submitted AFTER the ignore file was created. If you added files to the depot before, you would need to manually remove them and reset the file permissions to make them writeable. This is because if certain files are set to read-only, it will cause issues with certain settings and config files that are used by game engines and are specific to each user.
What to Do When Something Goes Wrong
So, you know that to work on game engine assets with P4, you need a typemap and an ignore file. But sometimes, things do not go according to plan. Let’s review some common errors.
Unable to Save Files
For example, you are working in Unreal, and you get an error that your Derived Data Cache is not setup correctly. This could happen because you didn’t have your .p4ignore file setup before adding files to the depot, which caused these files to be set to read-only. You will need to remove those files from the depot and then manually enable write permissions on your OS.
Builds are Failing with Permission Errors
If you are trying to build and it says that you do not have permissions, there may be some intermediate files that need the +w modifier on them so the engine can overwrite them when doing a build. You will need to set your typemap and manually change any previously submitted files to always be writeable.
Game Engine or Visual Studio Settings Are Not Saving
If your game engine or Visual Studio settings aren’t being saved, or they are changing to someone else’s, then certain user-specific folders and files are not being ignored. Be sure to check the p4ignore examples and then remove those files from the depot and reset their permissions on your OS.
Not a P4 Customer?
Discover why so many customers are using Perforce P4 for version control.