April 5, 2011

Evidence-based Reasoning and Forensic Debugging


In The Adventure of the Blanched Soldier, one of the stories in The Case-Book of Sherlock Holmes, Holmes describes how he goes about solving problems:

"That process," said I, "starts upon the supposition that when you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth. It may well be that several explanations remain, in which case one tries test after test until one or other of them has a convincing amount of support."

Doyle, of course, didn't invent the process of evidence-based reasoning; this approach has been a prominent part of our lives at least since the mid-1600's, when thinkers such as Christopher Wren, Robert Hooke, and the other founders of the Royal Society popularized the notions of experimental science and the open sharing of knowledge and discoveries. (If you're interested in these events, may I recommend Neil Stephenson's Baroque Cycle trilogy.)

I was thinking about this the other night, while my wife and I were watching the popular CBS television series NCIS, and I was reflecting on the necessity of maintaining the proper discipline when you are trying to solve a problem.

You may think you know the answer, you may believe you are right, but you have to always be careful to carry out your reasoning as Holmes describes, and don't allow yourself to conclude that something is true until it has been proven, not just conjectured.

At work, we spend a certain amount of time doing what I call "forensic debugging", which others might call "reading core files and error logs." A core file is an amazing thing; you can (and should) approach it just as Holmes would approach the scene of the crime. In the core file lies the truth, if you know how to interpret it. You must be thorough, patient, and careful. Take notes, check and re-check your findings, frequently review your analysis to ensure you have not jumped to conclusions.

Most of all, remember that it doesn't matter what you want the answer to be; it only matters what the actual answer is. Pay attention to the evidence, and you will figure out the answer.