Mark Gilbert's Blog

Science and technology, served light and fluffy.

"Testing after Unit Tests"

In one of Scott Hanselman’s more recent installments of Hanselminutes, he interviews Quetzal Bradley of Microsoft.  The interview, titled “Testing after Unit Tests“, focuses on the epistemology of unit testing and code coverage – what knowledge do we really gain by these practices and metrics?

Bradley makes several very interesting observations during the session, but one of the most enlightening was about code coverage.  He says that people like to think of “code coverage” as a positive metric.  For example, “my unit tests cover 60% of my code”.  The real value in that number is not the 60% of the code base that is covered, but rather the 40% that isn’t.  In other words, code coverage as a metric does a good job of telling you what you have not yet tested.

Bradley also makes the point that code coverage doesn’t guarantee that the code that is covered by tests is bug-free (he even gives the example of one of his past projects where the client found a problem in a section of code that was covered by one or more tests).  You can have several tests covering the same function, and therefore may achieve 100% code coverage for that function, and still not have all of the possible scenarios for that function covered.  Bradley describes a metric called “state coverage” that measures the number of scenarios, or states, that the tests cover.  Ideally, he says, we would aim for 100% state coverage in our tests, but usually that’s not realistic given time and budget constraints, so we need to prioritize our testing.

All in all it was a very valuable half an hour.

March 23, 2008 Posted by | Agile | Comments Off on "Testing after Unit Tests"