Beware Code Coverage Metrics

Code Coverage

Yesterday during our morning huddle one of the developer’s mentioned that on the story he was working on he had achieved 97% code coverage and this got me thinking a bit about something that’s been on the tip of my mind for awhile. Now, I cannot speak for the developer at my huddle because I haven’t seen his test and I can assume that they’re well written, but I often have heard through my career reports of good code coverage and have seen instances where code coverage was used as a bad metric in the worst possible way.


I’ve seen projects where 98% code coverage was bragged about, only to look at meaningless test cases that don’t even begin to illustrate the behavior of the system under test at all… in some cases even huge mammoth 300 line integration tests that, through pure luck, exercise the entire system functionality and achieve 80% code coverage alone on a 15,000 line code base. Additionally, I’ve seen people in the past run a tool like Clover or Emma and find areas not covered and write tests to exercise that functionality.

In such cases, I can comment out method bodies, rerun said tests, and still hit green bar even though my system contains major defects that, if deployed to production, can cost my stakeholders possibly thousands or millions of dollars. Where the frak is the benefit in that!?

If you’re guilty of the previous two sins, you’re doing it wrong. Unit Tests are NOT “tests” in the classical sense… they instead should be used to drive your design, to help you think about how the system you are writing should work, to illustrate functionality. When you use coverage tools to achieve code coverage for code coverage sake, you’re benefiting no one.

Code coverage is a side effect of well written examples (or tests as you might call them) not a metric that indicates the tests are well written. So don’t focus on the test coverage as much, it should naturally occur as a result of good examples and well fleshed out scenarios. You can use the tool to find areas not covered, as is often the case in legacy systems, but you should take proper care to ensure that when you put that code under cover that you are really describing it’s behavior well and not just exercising it.

If you ever find yourself running a code coverage tool after writing fresh code, step back and realize that you’re doing it wrong.. my suggestion is if you run a code coverage tool and find parts of your freshly written code aren’t covered, delete that code and try again. It might even be true that your class doesn’t even need that code anyway, because you never did prove the need for it to exist with a well defined example. ;)

  • Dan Watt

    Good article.

    Of course, you are the guy who wrote AssertGettersAndSetters ;) (though, I will admit that sadly that was because someone didn’t ‘get’ what beans are)

  • http://blog.james-carr.org James Carr

    Thanks Dan. Of course I wrote that utility only because someone used the code coverage metrics to attack the team. Luckily people learned since then. ;)

  • http://testovanisoftwarueng.blogspot.com/ Anna Borovcová

    It is true. Statments code coverage is a side effect.
    But you write about code coverage and mean only statments code coverage. It is common simplification.

    May be, you will find interesting my excercise about code coverage on
    http://testovanisoftwarueng.blogspot.com/2009/10/problems-with-code-coverage.html

    I hope my english is not too bad in this article.

  • Pingback: Unit Tests are Tests