May 27, 2009

TestTrack API Performance

Helix ALM
I recently got a call from our Support manager about a customer who was upset that TestTrack "wasn't working and was performing horribly" and they were not happy with us. Support had already confirmed that it wasn't a functionality issue so there wasn't much else they could do. They asked me if I would I be able to at least review the customer's setup? So I took the call and 2 minutes into it I knew there was something completely out of whack with their implementation. First, the company was a supplier to BMW and the "horrible performance" was in exchanging data through the BMW defect exchange interface. We've done several implementations of exactly that interface over the past couple of years without any performance issues. Second, they had spent 6 months (!!) trying to figure out why it wasn't working. They had contracted the integration work out to a local consulting company who had no TestTrack experience and the contractor had given up, sure that it was our fault. On average, it takes us a couple weeks to implement a project like this so I knew something was totally out of sync. The customer had log files from their recent testing and was more than happy to send them to me. The performance issue was caused by the fact that a simple query of the defect list was sending back 9MB of data, for something like 500 issues. When does 500=9MB? When you don't give any thought to performance. Depending on your TestTrack configuration, there can be hundreds of pieces of data associated with a defect. Think about the fields on each event in your workflow and you can see how the numbers multiply. In this  case, there were almost 300 in the customer's TestTrack setup and every field was being retrieved so 500 * 300 * XML overhead=9MB. // This is bad, you're fetching too much data CRecordListSoap list = engine.getRecordListForTable(l, "Defect", "", null); I was skeptical, I've written a lot of SDK applications and I've never needed more than 10-15 fields; usually it's more like 5. I called the customer, explained the issue, and we both agreed that not all 292 of those fields were needed. They improved the query logic, re-ran the tests, and everything started working. // This is much better, you only need a few pieces of data CTableColumn[] atc = new CTableColumn[2]; atc[0] = new CTableColumn(); atc[0].name = "Number"; atc[1] = new CTableColumn(); atc[1].name = "Status"; CRecordListSoap list = engine.getRecordListForTable(l, "Defect", "", atc); With great power comes great responsbility, I learned that from Spider Man and it applies to the TestTrack SDK as well. So be smart - there's a lot in the SDK but you rarely need all of it at the same time. Oh, and read the TestTrack SDK Best Practices guide before you start.