How to Reduce the Technical Debt of 1000 manual test cases?

Question

I want my team to move away from manual testing as much as possible. As part of that I want us to automate around 1000 manual test cases. Some of these tests are for legacy features, which don't even have dev owners. There is no one in the dev team who understands what these tests do and if we even need them. Many of these tests don't even find bugs, but the release engineers expect the QA team to run these tests because this is what they were doing for a while.

I do not think blindly automating 1000 tests is the right thing. How to engage devs also to solve this? Any suggestions?

Answer 1

There are a number of considerations.

You should avoid thinking of 'x number of test cases' means 'x number of automated cases'. As attractive as automation is, it is very limited in what it can do effectively, reliably and in a timely fashion. Some things it will be able to do better, others not as well. Humans do not have 'intermittent failures' - or at least not the same kind ;) so a different approach is needed.

These are some of the considerations you should have:

• Think about automated UI tests... along with back end Unit and Integrated tests that will run in seconds and minutes instead of minutes and hours.

• Avoid testing different data combinations through the UI

• Have Smoke tests at both UI and backend levels for extremely fast feedback in a few seconds as code is changed.

• Have Happy, Sad and Optional types of tests to cover the different scenarios

• UI tests are flaky. Everyone discovers this over time. We do not control the asynchronous devices (browsers, mobile devices, etc). and their asynchronous nature usually leads to failures over time

• Ask for a review group to go through the cases that are not understood. Management need to make sure that time is allocated (at a high level) to do this. If the task seems too overwhelming, try tackling 10 a week.

• Examine what actual bugs have happened recently and focus testing on that to add real value and not just test for testing sake (like those old tests you have inherited that no-one know what they do)

• Make sure application engineers, automation engineers and release engineers get together and have a good shared understanding instead of remaining in their own silos.

Think about how you are going to operate going forward on a daily and weekly basis. Clearly the development process will need to change to accommodate new approaches. This should start now and not wait for the 'cleanup' to be done first. Figure out how to create appropriate tests at all levels for new and changed functionality. Otherwise the tide will return and wash away the benefits you started to receive from your temporary sandcastles of automated testing.

Also buy some books, for example:

• Agile Testing by Lisa Crispin and Janet Gregory (my 'bible')
• BDD and TDD (various books)
• Humans vs. Computers: https://leanpub.com/humansvscomputers

Answer 2

Here's how I'd approach this:

Analyze and rank the test cases by risk - First, I'd go through and analyze the manual test cases, looking to rank them in a way that helps decide on the automation approach. Some of the factors I'd consider are:

• Impact - What would happen to a customer if this test case were to fail in production? A simple 1 to 5 or 1 to 3 scale is enough - you don't have to be super-detailed on this. I typically work on something of the order of 1 = unusable, 2 = it can be worked around but is a major pain, 3 = the workaround is a nuisance, 4 = it's irritating but it doesn't stop anything, and 5 = who cares?
• Probability - Making this decision will take a bit of data diving in your bug reporting tool. Again, a simple ranking will give you a broad measure. The more often the test case fails, the higher the ranking.

The combination of impact and probability will give you clusters of test cases grouped from most likely to fail and give you problems to least likely to fail and give you problems. In essence, it's a modified risk analysis.

Start with the highest risk group and analyze for automatability - Next, I'd take the highest risk group of test cases, and analyze them looking at these factors:

• What would it take to automate - If it's going to take hundreds of hours to get a flaky automation, it's better off handled as a manual test case. On the other hand, if it can be quickly and reliably automated, it should be.
• Can dependencies be isolated - Manual tests will have dependencies, but a unit test or other form of automated test could potentially reduce the dependencies and test the key functionality.

No matter how you rank them, this is very much a subjective assessment that will depend on how your application code is structured, the strengths and weaknesses of your teams, and the tools you have available to you.

Your best case scenario is that the functionality covered by these tests is so rarely used and so rarely breaks it's not important to test it at all, manually or through automation. You're not going to get support for a decision like that without hard numbers, which means the analysis and some guesstimates on numbers are essential.

Include the time it takes to manually run the test cases each release cycle - and don't forget to note that this is time your testers can't spend looking for other issues in the code.

Consider a long-term approach - Possibly the simplest approach you can work on is to work with your development team to add unit tests around any module in the code that lacks them whenever they work in one. That way each time a bug is reported against a legacy function, that function will get unit tests built as part of the correction. Gradually this will improve both developer understanding of the older code and your automated test coverage over time.