March 28, 2012

What is the Role of QA at Perforce?


perforce QA

I think everyone understands that QA is the organization responsible for testing our software. We're also responsible for reporting on the results of that testing, i.e., providing an assessment of product quality based on the number and nature of the defects that we uncover prior to release and that emerge later as customers use our products in new and unexpected ways.

So, on the one hand, we're partners in the development process, working closely with developers to find as many problems as we can before a product is shipped. 

And, on the other hand, we're working with the Release team to protect our customers against any defects that might slip through the "creative chaos" of development and into the released product. 

These two requirements -- the need to thrive in the helter skelter development environment, and the need to be a rock of stability in the final stages of production and release -- pull QA in opposite directions. How can we reconcile this?

The answer lies in finding the right blend of exploratory and automated functional testing, and sprinkling in a dash of "test by change".

Exploratory testing, as the name implies, is a journey into the unknown. There may be maps, in the form of requirements documents, functional specifications, user stories, and/or source change submissions, but a tester embarking on exploratory testing is going where no one has gone before. The code he or she receives from a developer might or might not exactly match the specs. If it does match, it might do exactly what it's supposed to do, but might fail spectacularly -- or in a more subtle way -- if someone uses it in a way that its creator didn't anticipate.

Exploratory testing is done as new functionality emerges in the code, or whenever existing functionality is modified. Thanks to the "change review" feature in Perforce, testers can monitor any part of the source code for a product, and can immediately respond to any change that a developer submits.

In addition to exposing the weak spots in a new piece of code, exploration gives the tester a way to learn how to use all of its features. This provides the basis for developing automated tests.

If exploratory testing is the journey to a new land, automated functional testing is the building of a settlement once we've arrived there. With  the knowledge gained during the exploration, testers map out the pathways through the code and develop scripts that exercise specific features, or that simulate the behavior of a customer using the product.

Those scripts allow test scenarios to be replayed again and again, quickly, and across a whole range of environments and operating systems while development work on the next set of features proceeds.

In Agile development, it's critical to have automated tests in place as early in the process as possible, but, regardless of the development methodology used, it may not be practical to create complex automated functional tests until the functionality has solidified. If the behavior of the product is shifting, as it often will while usage scenarios are being fine-tuned, those changes will tend to break the tests, and this can result in lost time and extra effort spent rewriting them. This is especially true for graphical products like P4V.

There is, however, a kind of automated testing that happens at the "bleeding edge" of code development, and that is less susceptible to changing functionality. This is "unit testing", and its focus is on the smallest bits of coded logic in the system, the individual bricks that make up the walls that product features are built upon.

Since unit tests are relatively stable in the face of changing functionality, they can be put in place very early in the process. In fact, unit tests are sometimes written before the code, itself, using a practice called "test-driven development". The tests fail until the code is correctly written, and then they pass unless and until a coding error is introduced.

So, given the competing needs for automation in Agile methodologies, "creative chaos" in development environments, stable functionality in support of automated functional tests, and solid, stable releases, what can be done?

At Perforce, our approach is to:

  •   Encourage coders to write unit tests as, or before, they write the code.
  •   Put testers close to the coders so they can explore functionality as it emerges.
  •   Develop automated functional tests as each feature solidifies.
  •   Run automated functional tests "continuously" against production-level builds.
  •   Watch developers' change submissions, and re-test accordingly.

Exploratory testing early in the development process finds most of the bugs that slip past the developers. Continuous builds, which run whenever a developer checks in a change, verify that the code still compiles, and automated test suites running after each successful build provide a guard against regression.

A "stress and interoperability" test cycle when all of the coding is complete provides a final assurance that customers will have a positive experience when they install and deploy a new release.

Interested in learning more about Agile development? Check out the 'Agile Adoption: Convincing the Skeptics' webinar with Bill Baffy, where he discusses the Agile adoption process from the viewpoint of a rookie ScrumMaster.