Relationships Viewer: Development and QA
Foreword by QA Manager Archie Mitchell
I felt like I was hit by a ton of bricks while attending the 2011 StarEast Conference in Orlando, Florida last year. I was sitting in a huge conference room with thousands of other QA managers and professionals listening to the keynote speaker. In the middle of his speech, he says, “Test by Change is a known testing methodology”. For some reason, I had to contain myself from jumping up in the middle of this crowd and screaming "PERFORCE."
After mulling the statement: “Test by Change”, I thought, who knows about changes better than Perforce? In fact, we use this methodology at Perforce to test our software. When a developer makes a change to the code, QA does a verification of the change. But unfortunately this is not an automated process; this is where the light bulb went off in my head. What if Perforce created a test utility that all Perforce customers could use to automatically test by change? What would Perforce need to provide? And this is where the story gets interesting.
After returning from my trip, I presented the idea of a Test by Change utility to a few peers. Everyone I spoke with could see how useful a utility would be for our customers as well as internally. At this point, I had a lot of different ideas about what I would like in a Test by Change utility. I decided to run this idea by James Creasy, our Director of Product Technology, to see what he thought about the idea.
Before getting out of his office, he had already drawn up a prototype on his white board of some of the things developers and testers might expect from a Test by Change utility.
Archie’s timing was perfect; I had been just been looking for a project for Product Technology’s summer intern, Andrew Lau. Andrew was able to turn my white board design into a running proof-of-concept during his 8 week internship here at Perforce. (Andrew has since joined Perforce full time, and works in QA on automation programming.)
The rest of this article describes the proof-of-concept web application that explores and displays the core information I thought would be needed to effectively support Archie’s vision of a “Test by Change” paradigm, backed by Perforce’s stored history of development changes and defects in the code base. Archie now has a tool that will help determine the feasibility of his idea.
White Board Design
Here’s a transcription of what was on my white board:
What we see here is a map of the relationships between many of the major types of objects that Development and QA jointly work with. Words in italics indicate a method of obtaining the relationship information.
For example, Development and QA at Perforce often communicate about code defects using the Perforce Jobs system. Jobs define collections of changelists as well as a relationship between the defect description and the changelists (specifically, that this collection of changelists fixes the described defect.) The relationship is revealed using the p4 fixes command. Similarly, changelists define collections of files via a list of depot paths. This relationship is described by the p4 change command.
Note that Perforce already contains most of the information to connect and cross-connect the information of mutual interest between Development and QA. Information not already stored in Perforce can be deduced in other ways. For example, in our code base parsing the output of our automated Doxygen documentation process connects a given UI object names with the source files that define them.
Put all this together and you can theoretically discover which changelists are related to a test case, a UI object, or the relationship between a test case and a set of depot paths. It was Andrew’s internship project to turn theory into reality.
Creating a Relationships Viewer
Once the network of relationships was mapped out, the next step was to create a viewer that allows exploring those relationships dynamically. Andrew created a web application that draws on the output of Perforce commands, as well as static code analysis and other sources, to create an interactive display.
You might notice that the boxes in my white-board relationships map have simply been lined up side-by-side.
A difference from the white board drawing is the viewer application automatically filters the results depending on what the user has selected:
Now we can answer questions like:
- What depot files are involved in fixes to a particular UI object?
- What depot files have never been mentioned in a defect?
- What automated tests are affected by changes to a particular file?
- Has this visual object been implicated in any defect in this release?
- How does the list of depot files receiving fixes in this release vary from the last release?
- What products and which QA personnel are likely to be impacted by a change to a particular file?
Note that the accuracy of these answers will depend on the amount of meta data history you have to go on, and this technique can’t absolutely define every relationship (here we can blame Alan Turing and his accomplice, mathematics.) So although it’s more of a shovel than a scalpel, I hope it still might prove useful.
A Couple of Thoughts
We could remove or suppress the UI portion of the viewer application and make it into a service to be called by other tools. These tools could query this service to answer questions about relationships between these objects in support of a "Test by Change" QA paradigm. For example, an HTML Stream Graph could be extended to display information on what automated tests are likely to be impacted by a particular depot file check-in.
Another thought: When I was a developer, I noticed the amount of time and effort the separate Engineering and QA departments expend communicating back and forth about these shared objects. This project sparked some thoughts on that subject, which I’ll make the subject of a future post; stay tuned!
What's Your Process?
What might a relationships viewer or service reveal in your development process?