August 3, 2010

Bringing the power of HTML5 to P4V

Flexible Workflows

Genesis

It was back in early 2007 that I first had the idea of using HTML in P4V. I was coding the Home Page of the (then new) Administration Tool, and thought “Why am I coding an HTML-like interface in C++/Qt, why not just use HTML?” In this post, I describe the long road that lead to the 10.1 release of the Perforce JavaScript API for the Visual Tools.

Although it was clear to me that HTML was a natural choice for user interface development, there were numerous problems to actually creating a usable implementation- not the least of which was lack of an HTML rendering engine in C++!  I looked into porting the rendering engine from an open source browser, but decided that was too much work to be practical.

Fast forward to the spring of 2008, when Qt, a C++ cross-platform development library we use, released a WebKit integration in version 4.4.

(WebKit is the HTML rendering engine that powers Google Chrome and Apple Safari web browsers.)

Now the dream of using HTML to create interfaces in P4V became a reality, and I checked a tiny prototype into my dev branch. A small group of HTML visionaries formed: Jason Gibson, Bill Baffy and Michael Bishop.

Three Problems

Although we created a working HTML prototype of the Administration Tool’s Home Page running inside P4V, three problems became clear: first, without access to data in the Perforce server, the interface could not be very useful; second, where could the HTML files be stored?; and third, how do we know which HTML file goes where in P4V?

To solve the first problem, a key decision was to create a JavaScript API to P4 commands rather than an API to P4V’s object data. This protects the HTML from complexities in P4V and allows the HTML to be much more portable.  I also chose to implement the P4 command line API instead of a classic Perforce derived API format (e.g. P4Ruby). This eliminates the requirement for special JavaScript P4 API documentation- you simply use the existing command line reference documentation, as well as the command syntax you are already used to.

e.g. to list pending changelists of user “myuser”:

Command line:

 p4 changes -u myuser -s pending

JavaScript API:

 p4(“changes -u myuser -s pending”);

The second problem, where to put the HTML, proved trickier, and was not completely solved for almost another year due to other intervening responsibilities.

Once we had the idea of using an external file, loaded by P4V to define what HTML is used, it was a small but important leap to make that external file a JavaScript application itself- and further to expose the server JavaScript API to it. This meant that the external definition file, which we call the “central settings file”, can make decisions using the results of Perforce commands and anything on the intra/internet to determine what HTML components are used!

Now we had a powerful and flexible mechanism to determine “what” HTML is used. To determine “where” in P4V, we introduced the idea of “extension points” in P4V.

(Extension points come in two flavors: “extension by addition” and “extension by replacement”. “Extension by addition” creates new HTML windows.   “Extension by replacement” substitutes an HTML application in the place of an existing P4V window.)

The text keys that define extension points are the link between the central settings file and P4V: P4V asks the central settings file about a specific key at the point it is creating new tabs or about to display, for example, the submit dialog, and the central settings file can opt to return the name of an HTML file to use at that extension point. The name of HTML files used at extension points we call “entry points”.

(Entry points are analogous to the index page of a website- they are the first page you see, but can be the doorway into other pages.)

Now we can define generic extension points during P4V development very quickly- simply define the key name, ask the central settings file when creating or using that window, and load the HTML of the entry point into the WebKit window.   Best of all, the externalized central settings file lets us customize and extend an installed P4V endlessly.

Now we faced a new problem: we supported only local files as entry points, which leaves to the customer the unacceptable problem of getting the files into the right locations on each hard drive.

The solution was to enhance P4V’s processing of entry points to encompass four different syntaxes:

  1. Local
  2. Perforce Depot
  3. p4:
  4. http:

Depot paths are loaded by using p4 print to load the contents of a file on the Perforce server directly into the WebKit engine. Extending Qt’s Network Access Manager, we added a P4URI type to extension points. This allows the central settings file to return entry points that are in a different Perforce server. This also enabled using an http: entry point. In essence, P4V now sees a Perforce server as if it was a web server.

So now we knew, given a central settings file, how to get the right HTML entry points to the right extension points in P4V. One perplexing problem remained- how does P4V find the right central settings file? I rejected hard-coding a depot location- too fragile. Storing the central settings file in a counter was considered, but that requires hard coding a counter name, and counters can be modified by non-super users. I needed something that only a super user could change, but any user with read permission could see. The answer was in the protect table: only a super user can modify it, but a user can read lines from it that match their user using “p4 protects”. I could put the location of the central settings file into the depot path field of the protect form; luckily it would also take a local path. P4V simply runs “p4 protects” and reads the central settings file location from it. This was a powerful solution- no hard coding of locations or key names that could collide with user identifiers, great security, and as a freebie, full access to the users and groups functionality in the protect table. The administrator can conveniently assign central settings files, and thus, GUI extensions, via the existing group structure.

Success!

Now we have a system where the central setting file is stored and versioned on the server.  The HTML applications, now called “applets”, are stored and versioned on the server and transparently delivered to P4V and P4Admin all under central control.

2009 widely saw the recognition of JavaScript as the de facto language of the web, as well as some heady performance improvements over just a few years before. The SquirrelFish engine in WebKit is state of the art, blazing fast, and even includes Just-In-Time compilation on some platforms. This is not your parent’s JavaScript!

There was a formative event that pushed the importance of this project in my mind from interesting to essential- the 2009 Perforce User Conference.  I saw several examples of complex, sophisticated tools custom built by companies to solve the same sort of problems I was tasked with for my internal development. Although they were all custom built, they all had one thing in common- they were all web (HTML) based, and at once I realized it made a lot more sense to allow easy integration of these existing tools into our tools than to built them myself in C++. Around the same time I became aware of the dizzying array of JavaScript libraries supporting almost any use case or visualization I could imagine.

One more surprise

There was one last surprise, and it was a doozy. I had essentially co-opted the power of the browser for doing general application development, but what can a browser do? The surprise was that the WebKit I was using for simple forms and dashboards supported HTML5. HTML5 contains amazing features like a high performance two-dimensional drawing surface, direct video and audio support, geolocation, local storage via a built-in SQL database, web sockets, offline support, and multi-threaded capabilities. Not convinced? Or a Photoshop replacement? http://mugtug.com/darkroom/ Or a Quake II port to HTML5? http://googlewebtoolkit.blogspot.com/2010/04/look-ma-no-plugin.html Keep in mind these don’t use plug-in’s; it’s JavaScript running in the browser that’s doing the heavy lifting.

It was just this year, 2010, that the industry began waking up to the power of HTML5, and here it is, already integrated in P4V and P4Admin in the 10.1 release. What will you build with it?