Mathematical / statistical relationship between tests and bugs?

Question

As project manager, I would like to convince higher management to invest more time for development of unit tests for existing methods. My understanding is that unit testing is foundation/pre-req for any meaningful functionality/integration testing and investment into functional/integration testing without reasonable coverage of unit test would not be wise.

Basically our situation is that after several years of development, we had nearly 0% of unit test (measured by how much code was covered by test).

2 months after forcing developers into unit testing, we have like 2% (unit test written only when new method is introduced or when any method is changed). With current rate of work, it would take ages to cover any reasonable % of methods with unit testing.

Is there any publication/graph/research/case study, which I can show to them and demonstrate relationship between probability of introducing hidden regression bug (ie, bug detected by end user and not by unit test) vs unit test coverage.

I know, that unit test coverage is bad metric, however is currently best understandable by higher management and I'm in situation, when there is almost no documentation covering all use-cases/required functionality, where I could match integration/functional test against requirements.

Answer 1

There is no correlation between the number of tests in a codebase and the number of bugs found in production. This makes sense if you think about it. If the tests are created solely to pass some "testing quota", there is no guarantee that they are actually well constructed tests. Also, if the feature is implemented incorrectly, the tests will most likely be written incorrectly because they reflect the creator's understanding.

That being said, unit tests are considered the best type of tests. They are designed to test one unit of work which is the smallest domination. Hence, they run faster than other forms of testing like integration testing. Unit tests are are written earlier in development while integration and end to end tests are written later. They therefore have a greater timeframe to bring value to the organization.

Test have other value other than catching bugs. Usually code which is too tightly coupled or has too many dependencies is difficult to write tests for. If unit testing is a requirement, the code generally gets simpler because it makes writing tests easier. That makes the codebase cleaner and less complicated. That reduces confusion and misinterpretation of what the code is supposed to do which results in less bugs. Also makes refactoring easier which speeds up development time.

Tests also make it clear what the intent of the code is. For instance, if you have an offshore team which sometimes misinterprets English instruction, giving them test cases makes it clear what the code is supposed to do. Same thing for on-site newly hired developers. It serves as both requirements and documentation.

A proper testing suite will also allow for greater amounts of build automation. Which reduces deployment times. Which allows you to get more features out the door in shorter timeframes.

Answer 2

There is no law of physics that guarantees a quantifiable payoff from writing unit tests. In fact, it's even possible to unit-test in such a way that your quality gets worse! Rather than attempting to find evidence in the literature, you might try a three prong approach like this:

1. Get agreement that you have a quality problem now and that developers need to be involved in addressing the problem. If management won't agree to that, consider finding a new job.
2. Show evidence that unit testing is a common practice among professional software developers. (That shouldn't be hard to do.) Since it's a common practice, it is a reasonable thing to experiment with.
3. Establish some quality metrics that you can monitor to determine whether quality is improving.

If quality doesn't seem to improve, you might decide that you need to try something else. It is even possible that you might need to do something else first before you adopt unit testing. But if you try something else, you should still agree on some quality metrics that you can monitor to determine whether quality is improving.

Note that it is easier to show that quality is getting better than to prove that unit testing is the reason quality is getting better. I once worked at a company where quality improved after we added a more rigorous testing process. It was never clear whether the new testing process caused the improvement or quality just got better because we slowed down the rate of change. In a sense, it didn't matter.

Answer 3

Your situation is very familiar and very common. I've seen it at many companies. Right now it is most prevalent in the US financial industry which has been 'discovering' Agile over the past 2-3 years. Given that they are the prime example of Command and Control to meet quarterly earnings its a very difficult journey for them.

The 2% figure is pretty good. I would focus on treating it as an investment. One that will grow over time the same way compound interest does. At first testing is incredibly hard. It's a new skill. Mocks and Stub are foreign notions and have to be built. Learning, learning, learning. After a while it will become easier. So expect your growth to be like this:

                           |
                           |
                          ||
                         |||
                       |||||
                     |||||||
                  ||||||||||
              ||||||||||||||
       |||||||||||||||||||||
||||||||||||||||||||||||||||

instead of what you fear which is more like

                                                               ||||||
                                                        |||||||||||||
                                                 ||||||||||||||||||||
                                          |||||||||||||||||||||||||||
                                   ||||||||||||||||||||||||||||||||||
                            |||||||||||||||||||||||||||||||||||||||||
                     ||||||||||||||||||||||||||||||||||||||||||||||||
              |||||||||||||||||||||||||||||||||||||||||||||||||||||||
       ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

Ultimately I recommend you focus on high quality code, good development practices (pairing, sharing, constant learning, lunch and learns every week, etc, etc.) and lots of tests. Buy and read lots of books. Hire programmers who are TDD and BDD evangelists.

Getting your company to value high quality code is very wise for the company future. However quality and making the case for it takes a lot of work by a lot of people working as a team.

I recommend "Agile Testing" which lays out a good plan for anyone doing agile:

Answer 4

Regarding the coverage issue - my team and I work with C# and use Typemock. It's a unit testing framework that has a tool called "Suggest". You can activate this tool on your code and it will suggest tests for it, and help you reach high code coverage in short time. About convincing management - you can read this article.

Answer 5

Yes, although not with the results you would expect. Research of actual cases showed no or low significant correlation between unit tests and quality. See here (I’m on mobile and cannot quote):

Mythical unit test coverage

And

on the relation between unit testing and code quality

Answer 6

How about taking production issues log and directly linking to how many of them could have been naturally covered in unit test layer? Ex. invalid data inputs.

In one of projects, we implemented a similar metric to research and understand that in the absolute absence of unit tests how much issues are coming back from users in production which could have easily be baked in a unit test suite.

Answer 7

There is a study that found a correlation.

Code coverage and test suite effectiveness: Empirical study with real bugs in large systems

[...] there is indeed statistically significant correlation between code coverage and bug kill effectiveness. [...] For HTTPClient, the correlation is moderate for both statement and branch coverage. For Rhino, the correlation is strong for both statement and branch coverage.