Accumulated mass of tests - how to work with business to remove / refine?

Question

Test object: a big monolith application (~500k loc) developed in Java for the last 15 years. Big and (probably overly) complicated backend + web frontend. There are many business processes implemented in the app. Currently there are 10 teams working on it.

Tests: there are unit tests and integration tests implemented. Code coverage is at around 70%. There are also MANY automated system and e2e tests implemented in a commercial test tool.

Issues:

1. there are way too many system and e2e tests. They were added over the years as at first there were no other tests being written. Only after a while someone thought of adding Java-based unit and integration tests.

2. Apart of the code coverage of the unit und integration tests the test coverage is a big unknown. The knowledge about which functionality is tested by what system test is kept only in the heads of the testers. There is no real way to know which tests need to be fixed, when a new story is implemented, other than running the regression test suite and to look what broke.

Question: In my opinion we need to get rid of a big chunk of the system tests ASAP, as they are not maintainable any more, flaky and take waaay to long to run (~24h concurrently on 50 VMs). There are dead tests, that test removed functionality, many duplicates etc.

In order to do that first I would like to know the test coverage, to be able to tell which tests are useless and can be deleted safely.

I am looking for a way to determine/estimate the test coverage in such a project

The obvious way of connecting the test with the corresponding user story/requirement will not work, as for quite many of the implemented functions there are no documented requirements.

The monolith is right now in the process of being refactored into roughly independent modules. The current idea is sorting the system tests based on their belonging to these modules. Additionally the tests belonging to one module can then be sorted based on the business process they test inside that module. With this I should have database of tests broken down by the modules and then by business processes, and thus a rough test coverage estimation by process and module. This still does not convince me, as being the best approach though.

So does this make sense? How else could the test coverage be determined/estimated?

Answer 1

Extremely common problem and largely represents the 'state of play' at a great number of companies, especially though that have had a software offering for more than 10 years.

Moving from manual testing to automation is frequently misunderstood. The target is assumed to be 'automating the manual tests' and when there is just that understanding, chaos ensues.

In order to have effective development and testing today you need to take a high level holistic view. This needs buy-in (and pushing) from very senior management. This is a massive change to how everyone involved will operate on a daily basis.

You need a lot of buy by the business in for the new ways to perform Unit, Integrated, E2E, performance and usability testing. The business will need to determine if it wants a modern CI/CD system. If it does then testing has to change. This needs to be driven by the business

The business must wish to adopt a test pyramid and not a test ice-cream cone or hourglass.

If the business wants to be able to move faster, release more frequently and have LESS bugs then they will need to buy in to the concept that LESS tests can be better. The focius should switch to MTTR and MTBF. These mean Mean Time To Recover, i.e. bad stuff is ok if u can fix it real quick and the benefit of speed remains. MTBF is mean time between failure and can also help you determine stability.

The business must direct and re-direct the current manual efforts (and deal with the usual politics and turfdom that comes in any business) and philosophy. This requires very strong leadership to understand and promote the change and the ending of the previous practices. Manual testing as the backbone of software testing has to switch to manual testing being used for exploratory and usability testing - areas that were often given short shrift previously due to time and resource constraints and the simple need to make sure basic stuff works.

What I've seen is that this is often not addressed.
New layers are added and speed is simply not addressed. Ultimately the company moves slower. It may mask this with stats such as "we've release 56 new versions this quarter!". Often the 56 versions were essentially planned in advance and would previously have just been 1 change that took 3 months. This does not lead to the company actually benefitting duie to the massive work needed for the co-ordination of the 56 pieces. This despite all the stats and charts that management are presenting to executives showing them what a wonderful success agile has been- up and to the right! The foxes are guarding the hen houses.

Answer 2

You can use a tool like Sonar to perform static code analysis and help you get an idea of the code and test coverage of your monolith.

Having a large number of tests is of no benefit if those tests (and their results) are not trusted. If you run the tests and the "expected" result is for them to fail, they are adding no value and I believe, can be disabled.

I would suggest you disable/comment out portions of the tests and from there establish if they are needed. If they are needed and add value, then re-implement them, if not, then remove them permanently.

In an application the size you describe, this will not be a quick or trivial task to complete properly.