August 26, 2016

Advanced form triggers with P4Python

What's New

I haven’t done a post on P4Python for a while, so I thought it was time for an update.

Today I want to talk about triggers, particularly form triggers. Perforce Helix can be customised in many ways, one of which is triggers: programs or scripts executed by the Helix server on certain conditions and controlled by the triggers table.

There are many different kind of triggers but the most common ones are submit triggers (executed when a submit is started, in progress or completed) and form triggers (executed when a form – such as a change, client, job and so on – is requested or sent by a client).[1] It is the latter I would like to focus on today.

Trigger recap

 

A form trigger has five actions:

Name

Invoked when

form-out

Sending a form to a client

form-in

Receiving a form from a client

form-save

Receiving a form after the server checked the content

form-commit

After a form has been processed

form-delete

Before a form is deleted

 

The first two triggers may modify a form. To do this, the server will create a temporary text document with the form content (such as the client workspace or job) and then execute any trigger defined for this action, passing the filename into the trigger as a parameter together with other available parameters, as defined by the entry in the triggers table.

The trigger could be a simple shell script, a compiled executable or a script written in a scripting language such as Python, Ruby or Perl. For these scripting languages the derived APIs (viz., P4Python, P4Ruby and P4Perl respectively) are particularly convenient, because they offer the facility to parse the form in the temporary file and return a dictionary with its content.

 

Trigger details

 

We have here, for example, a client workspace trigger written in Python using P4Python, invoked via the trigger table in the following way:

 

           update-client form-out client “python client-trigger.py %formfile% %formname%”

 

The %formfile% and %formname% items are trigger variables the server will replace with the real values. I could then write the following trigger script:

 

            import sys, P4


            p4 = P4.P4()

            filename = sys.argv[1]

            formname = sys.argv[2]

            with open(filename) as f:

                        content = f.read()

            form_content = p4.parse_client(content)

 

The variable form_content now points to a dictionary with the names and fields of the client workspace. I can then use the script to, for example, set the default options:

 

            form[‘Options’] = form_content[‘Options’].replace(‘normdir’, ‘rmdir’)

            content = p4.format_client(form_content)

            with open(filename, “w”) as f:

                        f.write(content)

 

Very convenient, as it avoids parsing the content of the file myself and potentially replacing unintended fields such as the description field. So, parse turns the raw text file into a dictionary and format takes that dictionary and recreates the text file.

But, there is something odd about this script:

I have not specified the port or user of the server, nor have I called the connect method. If the script does not talk to a Perforce Helix server how does the script know which fields exists in a form, and how can it parse its content?

 

Form processing details

 

To answer that question, let’s backtrack a bit. For a standard (non-trigger) script it is quite common to manipulate a form, such as a client workspace or a change. The standard methods used to retrieve and save a form are P4.fetch_<form _type> and P4.save_<form_type>(), for example

 

            client_content = p4.fetch_client(“my-client”)

            p4.save_client(client_content)

 

What actually happens under the covers is that the server sends the script the client form as a text document and a so-called spec string that contains meta information about available fields in that client workspace document. The C++ API, on which P4Python is built, takes the spec string, stores it inside a local cache and then uses it to parse the form.

All this requires that I have a connection to the server to retrieve both the form and the spec string. But for form triggers I would like to avoid any connections from my trigger script if I can, since I then do not have to worry about logins and tickets.[2]

To be able to parse any forms without a server connection, all the derived APIs have the spec definition for every possible form compiled in. If you are really curious about how it looks like, look into SpecMgr.cpp in, for example, the P4Python code (here for change):

 

“change”,

"Change;code:201;rq;ro;fmt:L;seq:1;len:10;;"

     "Date;code:202;type:date;ro;fmt:R;seq:3;len:20;;"

     "Client;code:203;ro;fmt:L;seq:2;len:32;;"

     "User;code:204;ro;fmt:L;seq:4;len:32;;"

     "Status;code:205;ro;fmt:R;seq:5;len:10;;"

     "Type;code:211;seq:6;type:select;fmt:L;len:10;"

     "val:public/restricted;;"

     "ImportedBy;code:212;type:line;ro;fmt:L;len:32;;"

     "Identity;code:213;type:line;;"

     "Description;code:206;type:text;rq;seq:7;;"

     "JobStatus;code:207;fmt:I;type:select;seq:9;;"

     "Jobs;code:208;type:wlist;seq:8;len:32;;"

     "Files;code:210;type:llist;len:64;;"

 

This data contains the name of the field, its type (such as date or string) and formatting information for tools like P4V.[3]

 

Upgrade challenges

 

There is a catch: if you upgrade your server, and that newer release changes the spec (which only adds fields as we never remove them on upgrades), then you also need to upgrade your P4Python package to pick up the latest compiled spec definition, or your form-triggers will fail with an exception.

A good example was the 2015.2 server release, which added the new field “ImportedBy” to a change. Without an upgrade to P4Python 2015.2, any of your “change” form triggers using the techniques described above failed.

This is ... inconvenient, both for you, the user, and for us, the API writers. The process to update the compiled spec is automatic, but we would still like to avoid having to release a new version of a derived API if we did not change anything else.

So, in 2016.1, both the server and the derived APIs picked up a new trick. The server gained a new trigger variable it can pass into the trigger called %specdef%, which contains – you guessed it – the very spec definition we need.

P4Python as the first of the derived APIs acquired a new method, P4.define_spec(), that allows up to update the local spec cache (the other APIs will follow shortly).

So, now my new trigger can be invoked in the following way

 

            new-form form-out client “new-form.py %specdef% %formfile% %formname%”

 

The trigger script can then use these arguments:

 

            import sys, P4


            p4 = P4.P4()

            formspec = sys.argv[1]

            formfile = sys.argv[2]

            formname = sys.argv[3]


            p4.define_spec(“client”, formspec) # here our cache gets updated


            with open(formfile) as f:

                        content = f.read()

            client_content = p4.parse_client(content)

 

Your script is now independent of any future server upgrades beyond 2016.1. Note that you need to specify which spec definition you update (the “client” argument).

There is an additional benefit in this. Specs like clients or changes cannot be easily changed by you, our customers, which is why we could add them as compiled objects to the derived APIs in the first place. This is not true for jobs: they can be explicitly changed through the ‘p4 jobspec’ command, and many Perforce users make use of this capability to add new fields and status values.

It was therefore never possible to create a “form” trigger for jobs that does not have a connection to a Perforce Helix server. With the new %specdef% variable, you can do just that.

Too good to be true? Well, try it out. There are many other tasty new features in the server release 2016.1, and a few goodies thrown in for good effect in P4Python 2016.1 as well.

Any questions or feedback, send a tweet to @p4sven.

 

Happy hacking!



[1] One the command line this is when P4 will open up an editor with the form document.

[2] The processing, after all, happens in a trigger on the server machine behind the firewall.

[3] Look at ‘p4 help jobspec’ for some details on the data encoded in the spec string.