How would we know when we are over-testing?

Question

I've recently heard the idea of over-testing for the first time while listening to the interview of the coverage.py maintainer on the Talk Python to Me podcast show.

coverage.py has a new feature to track what and how many tests called a specific line in the code. A potential suggested use case for this feature is to look for the lines in the code that have been executed "thousands" of times during a test run indicating that these lines were over-tested and there is a potential to cut down on the number of tests - with a motivation to make a test run faster.

I'm a bit confused as I cannot imagine myself deleting tests because of the over-covering specific lines in the application code. Are there any other potential problems of over-testing?

Answer 1

We're over-testing when we are not adding value

It's easy to test to feel good but it's important to do testing that truly adds value as the business defines it.

This is a difficult thing to determine however as there are a number of factors to consider in order to determine both cost and value added by the testing. It is further complicated that some testing is usually percentage based risk aversion, e.g. if the risk of breakage is 2% it's worth testing X amount.

Factors to consider

• How long it takes to fix bugs
• The consequences of things breaking
• How production breakage is monitored
• The use of BlueGreen or Canary releases
• How long the testing takes. Time is money
• How much the testing costs in terms of people
• How much the testing cost in terms of machines
• The speed of change in the market by competitors

• How long it takes to roll back a change to production

• How isolated the various pieces of code are (ease of testing)

• How many unit vs. acceptance vs. integrated vs. UAT test exist

It is also worth calling out that over-testing is often specific to the type of testing. Given that too many well-written unit tests... is usually not a problem... there is often one specific situation:

The most frequent problem is too many UI tests.

This happens when strategies (intended or accidental) emerge to 'test everything through the UI'. This happens in organizations which have low quality or non-existent unit tests or where unit tests do not mock and stub dependencies and are thus integrated tests. It frequently happens when there are a number of data variations that can happen based on backend routines using data entered in the front end. There should be unit tests of the backend routines themselves, but when these tests do not exist or are not high quality many business again fall back to 'testing every combination through the UI'. When 'test every data combination through the UI' gets multiplied by number of browsers to tests X versions of those browsers X devices to test X number of versions of those devices X number of versions of the devices OS we want to check we are waiting for the universe to end.

The basic solution for too many UI tests is to seek to have a test infrastructure that leads to a good test pyramid for number of tests that looks like

          Manual
       Automated UI
     Integrated Tests
   Individual Unit Tests

For example a company might have the following numbers of tests

          Manual 10
       Automated UI 20 
     Integrated Tests 40
   Individual Unit Tests 375

Flat pyramids are more stable. Avoid pyramid Towers such as

        Manually 10
        Automated 12 
       Integrated 15
       Unit Tests 20

and make sure that quality unit tests are your first line of defense and are written before (or with) the associated code that is added or changed.

Answer 2

I've not, yet, listened to the podcast so the what follows here are my initial, unfiltered, tuppence worth thoughts on the simple question regarding over-testing implied by coverage.py output so far as I understand it.

Having scanned the doc set for coverage.py, I cannot find any mechanism for determining data variance over the executed code. The outputs from coverage.py are percentages for statement and branch coverage (full or partial). There are no references regarding the input data sets triggering these lines. Without reference to the data coverage, it is not possible to determine whether any given execution of any line is unique or not, rendering that test a candidate for removal.

Care has to be exercised with the output of any process generating only coverage metric for statement/branch coverage. It may be that, in spite of a line of code being exercised multiple times, a significant boundary condition, explicit or implicit, has not been tested, resulting in a false positive. A cast error resulting in an overflow condition is just one possible example which may go undetected without data coverage analysis being carried out alongside statement/branch coverage analysis.

It may be that you're testing for something pretty esoteric, such as a memory leak, in which case re-testing the same code with the same data would be understandable. Otherwise the time given over to re-testing a line of code under identical data conditions is likely to be better spent exercising alternate data conditions.

Over-testing can also lead testing into disrepute if we claim "We've run n000 tests which have all passed." yet the application is still found to have an unacceptably high defect rate.

Coverage.py appears to represent a useful additional tool to the testers toolset, especially given the ability to drill down from the output html to the underlying .py code marked up according to whether it is covered, missing etc. As with all tools though, it's limitations have to be understood and factored in when using it and, as we all know in testing/QA, there ain't no silver bullet.

Answer 3

---

Over-testing also hides defects in layers.

---

I discovered in test code reviews & in executions analysis that if we repeat a basic initial assertion in multiple transactional flow tests(maybe due to copy/paste) then on failing it fails all those tests and hides all the other subsequent defects which might be far critical than this one and more importantly for which those tests were originally designed.

So it is important as per my experience that every test just have a one and only one assertion which is the primary objective of the test. A test should only fail because of one reason for which it was designed.

To achieve this, we should analyse and remove all the redundant assertions from the tests and make sure that every assertion should be carefully placed inside a single test only. This will increase the effectiveness of a test suite in multitudes.

Answer 4

The biggest problem with over testing is maintenance of tests. First you are allocating resources to create the tests, then your tests require resources to maintain. Since your man power is finite, this will eventually reduce the amount of features you can produce. In extreme cases, test maintenance becomes a job, onto itself.