6

As project manager, I would like to convince higher management to invest more time for development of unit tests for existing methods. My understanding is that unit testing is foundation/pre-req for any meaningfull functionality/integration testing and investement into func/int. testing without reasonable coverage of unit test would not be wise.

Basically our situation is that after several years of development, we had nearly 0% of unit test (measured by how much code was covered by test).

2 months after forcing devs into unit testing, we have like 2% (unit test written only when new method is introduced or when any method is changed). With current rate of work, it would take ages to cover any reasonable % of methods with unit testing.

Is there any publication/graph/research/case study which I can show to them and demonstrate relationship between probability of introducing hidden regression bug (ie, bug detected by end user and not by unit test) vs unit test coverage.

I know that unit test coverage is bad metric, however is currently best understandable by higher management and I'm in situation when there is almost no documentation covering all use-cases/required functionality where I could match integration/functional test against requirements.

Thank you for any hints/advice.

3

There is no correlation between the number of tests in a codebase and the number of bugs found in production. This makes sense if you think about it. If the tests are created solely to pass some "testing quota", there is no guarantee that they are actually well constructed tests. Also, if the feature is implemented incorrectly, the tests will most likely be written incorrectly because they reflect the creator's understanding.

That being said, unit tests are considered the best type of tests. They are designed to test one unit of work which is the smallest domination. Hence, they run faster than other forms of testing like integration testing. Unit tests are are written earlier in development while integration and end to end tests are written later. They therefore have a greater timeframe to bring value to the organization.

Test have other value other than catching bugs. Usually code which is too tightly coupled or has too many dependencies is difficult to write tests for. If unit testing is a requirement, the code generally gets simpler because it makes writing tests easier. That makes the codebase cleaner and less complicated. That reduces confusion and misinterpretation of what the code is supposed to do which results in less bugs. Also makes refactoring easier which speeds up development time.

Tests also make it clear what the intent of the code is. For instance, if you have an offshore team which sometimes misinterprets English instruction, giving them test cases makes it clear what the code is supposed to do. Same thing for on-site newly hired developers. It serves as both requirements and documentation.

A proper testing suite will also allow for greater amounts of build automation. Which reduces deployment times. Which allows you to get more features out the door in shorter timeframes.

  • +1 for highlighting value besides catching bugs – semaj Jun 26 '18 at 15:25
2

There is no law of physics that guarantees a quantifiable payoff from writing unit tests. In fact, it's even possible to unit-test in such a way that your quality gets worse! Rather than attempting to find evidence in the literature, you might try a three prong approach like this:

  1. Get agreement that you have a quality problem now and that developers need to be involved in addressing the problem. If management won't agree to that, consider finding a new job.
  2. Show evidence that unit testing is a common practice among professional software developers. (That shouldn't be hard to do.) Since it's a common practice, it is a reasonable thing to experiment with.
  3. Establish some quality metrics that you can monitor to determine whether quality is improving.

If quality doesn't seem to improve, you might decide that you need to try something else. It is even possible that you might need to do something else first before you adopt unit testing. But if you try something else, you should still agree on some quality metrics that you can monitor to determine whether quality is improving.

Note that it is easier to show that quality is getting better than to prove that unit testing is the reason quality is getting better. I once worked at a company where quality improved after we added a more rigorous testing process. It was never clear whether the new testing process caused the improvement or quality just got better because we slowed down the rate of change. In a sense, it didn't matter.

0

Regarding the coverage issue - my team and I work with C# and use Typemock. It's a unit testing framework that has a tool called "Suggest". You can activate this tool on your code and it will suggest tests for it, and help you reach high code coverage in short time. About convincing management - you can read this article.

0

Yes, although not with the results you would expect. Research of actual cases showed no or low significant correlation between unit tests and quality. See here (I’m on mobile and cannot quote):

Mythical unit test coverage

And

on the relation between unit testing and code quality

  • Interesting read but that's overinterpretation. What they have found in Ericsson is that ". 100% coverage does not entail 100% tested code. ", which means that despite high unit test coverage there still might be bugs, which is expected, that's why there methods of testing like integration testing, exploratory tests etc. However, the question remains which method is more effective, where to put money? Perhaps, lowering unit testing standards (without excluding them completely) and investing more time and resource in other methods is more effective. – dzieciou Jun 29 '18 at 7:33
  • @dzieciou you are right about over interpretation, I suppose they were trying to make a point. The problem is that even if you find research material there is a good chance it will not be relevant to your case. At the moment I am struggling with a similar question, we produce more unit tests simply because they are easy to write and quick to run but I am not sure about their contribution to the future of the project. – Rsf Jun 29 '18 at 7:50
  • Software engineering is generally hard research domain, especially hard to make generic conclusions. I've been on the other side" we wrote too many end-to-end tests that are more fragile and harder to maintain. We moved them to integration tests -- better isolation, still test more than unit tests. But that's a good question: how do I know my tests contribute to the future of the project? – dzieciou Jun 29 '18 at 9:42
0

How about taking production issues log and directly linking to how many of them could have been naturally covered in unit test layer? Ex. invalid data inputs.

In one of projects, we implemented a similar metric to research and understand that in the absolute absence of unit tests how much issues are coming back from users in production which could have easily be baked in a unit test suite.

0

Your situation is very familiar and very common. I've seen it at many companies. Right now it is most prevalent in the US financial industry which has been 'discovering' Agile over the past 2-3 years. Given that they are the prime example of Command and Control to meet quarterly earnings its a very difficult journey for them.

The 2% figure is pretty good. I would focus on treating it as an investment. One that will grow over time the same way compound interest does. At first testing is incredibly hard. It's a new skill. Mocks and Stub are foreign notions and have to be built. Learning, learning, learning. After a while it will become easier. So expect your growth to be like this:

                           |
                           |
                          ||
                         |||
                       |||||
                     |||||||
                  ||||||||||
              ||||||||||||||
       |||||||||||||||||||||
||||||||||||||||||||||||||||

instead of what you fear which is more like

                                                               ||||||
                                                        |||||||||||||
                                                 ||||||||||||||||||||
                                          |||||||||||||||||||||||||||
                                   ||||||||||||||||||||||||||||||||||
                            |||||||||||||||||||||||||||||||||||||||||
                     ||||||||||||||||||||||||||||||||||||||||||||||||
              |||||||||||||||||||||||||||||||||||||||||||||||||||||||
       ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

Ultimately I recommend you focus on high quality code, good development practices (pairing, sharing, constant learning, lunch and learns every week, etc, etc.) and lots of tests. Buy and read lots of books. Hire programmers who are TDD and BDD evangelists.

Getting your company to value high quality code is very wise for the company future. However quality and making the case for it takes a lot of work by a lot of people working as a team.

I recommend "Agile Testing" which lays out a good plan for anyone doing agile:

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.