How to manage a lot of automated test cases

Question

We are a team of 8 QA Analysts. We have automated more than 3,000 test cases for our application. As this number grows bigger and bigger, we are having issues controlling tests that are already created and I think we have a lot of duplicated test cases. Not only duplicate, but also tests that are already included in another test (Test X does what Test Y already does plus other things).

I was wondering how you guys manage your test cases do avoid having duplicated or not useful ones.

We are using TFS and Microsoft Test Manager and our tests are automated using Selenium.

Answer 1

This problem hits everyone with a decent-sized application sooner or later.

Some of the things you can do to help manage your test cases more cleanly:

• Use a self-documenting system - if you're coding Selenium with C#, use the MS XML commenting and something like Sandcastle to publish the documentation that's generated on build to a common site. That way everyone in the team can search for test cases that do what they need before creating a new one.
• Try to string tests together in a sequence rather than include extras in another test - what I mean here is instead of having test A log on and perform action X, you have test A log on, test B performs action X (with a prerequisite of test A having been completed successfully) and so forth. This does introduce extra dependencies into your test runs but it keeps your test code base more DRY.
• Create library routines for common functions - this includes library routines for logging on, navigating through the application, and so forth - essentially anything that's stable in the application and/or used a becomes a library and you focus the actual tests on the aspects of the application that need more intense examination.
• Review and refactor - your automation code needs to be reviewed and refactored at least as much as production code, and regular reviews/refactoring will help to locate and clean out duplicates and unnecessary tests.

None of these are going to solve your problem. You're always going to have the tension between keeping your tests self-contained and keeping your maintenance load as low as possible. These are just some suggestions to help you identify where you want the balance between lower maintenance (DRY code) and self-contained tests.

Answer 2

Naming and separating out your reused parts (tests, keywords for maps). I call this "common" tests/maps.

1. If you already have alot of tests automated then this will be more tedious, but pulling out your unique maps and naming them appropriately will identify the actual controls that are being duplicate tested.
2. Test Naming conventions that follow the Functional area and then work flows for the tests and the page layout for the maps can allow you to quickly and easily identify which pages/controls are overlapping as well as which test cases are overlapping.
3. keyword variables for Map naming conventions so that you can see which maps belong to which pages and if they are being reused in mass or if they only truly exist on that page. Either way having a single keyword for a control that is duplicated all over the site, but maintains the same map value would allow you to attach to a keyword variable that exists in a single place and your maintenance on that map would be virtually eliminated.

After you have done the above the following should be possible which will make life so much easier.

1. Overlapping test case steps are taken out and form new "test cases" or functions. Then the previous tests will execute those steps in a test call instead of the previous steps. Similar idea to MTM "shared steps". In selenium terms these common tests become regular functions that are called and perform a set of steps instead of existing as a test in and of themselves.
2. Keywords are lined up with tests to get a functional workflow down to a functional control view of what is being tested in the application. Keywords are grouped where they are utilized which should hopefully be unique to test cases now. The commonly reused controls will most likely be in the common test cases from item 1.

This will cut your maintenance headache out and allow you to scale very easily. Your reporting will also align to your naming structure so you will be able to see where the problem areas are easier. When a test fails you will be able to track it to a control which should at this point exist in a single place and once corrected there all your tests will pass again.

This isn't a quick process, but the trade off is in maintenance going forward. I also have a custom report for automation tests and I can summarize functional areas and drill down to the individual control problems and see an exact error for each step, but it's up to you what you need and what's useful for your team.

Answer 3

It's possible you're tackling a symptom and not the root cause?

1. Do you have User Stories* and are they managed in some tool?

2. Do you have acceptance criteria for each User Story?

3. Do you formalize the acceptance criteria as Test Cases?

4. Do you "link" each automated test to its corresponding Test Case?

If you can answer Yes to all questions then all you need to do is to manage your User Stories and Test Cases. It's very unlikely a test case will fall into two different User Stories, and if it does there might be a valid reason for it.

Additionally, tools for handling User Stories and Test Cases are usually much more prepared to handle this information so you should be able to query for duplicates, etc., very easily.

If you answer No to some of the questions then I'd say that's the root cause of your problem.

Additional benefits of this approach:

1. The chances that you're testing something that's not important are greatly reduced because you only test what the Product Owner and team decided was important.
2. Documentation of the automated tests in the code is now just about including some link to the Test Case.
3. Any business person can easily review your Test Cases and deprecate/improve them.

*You can replace User Story with requirement or something else if you're not using an Agile process.

Answer 4

One thing that is often overlooked is to make use of your coverage tool(s) for more than ensuring that every line of code gets tested - if you configure your tool to produce a separate named output file for each test/test sequence then you can quite easily scan these for lines/blocks that are being tested in more than one test.

The other key point is - as with everything - documentation is key - the better the documentation the easier your maintenance will be.

Answer 5

Break out tests into the various kinds of test, i.e.

• unit tests that are written along with application code
• integrated tests that test dependencies
• performance tests that test both response and volume
• manual tests

and think about what is needed for your business, for your industry. A new startup with little funding will do quite different testing from Facebook.

Also, think about the testing pyramid:

        ui
     Integrated
 Performance & Load
Individual Unit Level

You should have a few high level ui cases but hundreds or thousands of low level unit cases (that mock and stub out actual services). In terms of test duplication between level, I actually expect ALL high level integrated tests to, in some ways, duplicate what is covered by low level unit tests. For example a user registering on the site and making a purchase might be one end-to-end ui test, however it will use a number of pieces of code that have unit tests to achieve this.

Time to run is also a factor. You should have unit test suites that run in under 5 minutes. Unfortunately many companies have test suites that run for 2 or 4 or even 24 hours. This means they now have technical debt that can cripple them if not addressed. Addressing this will mean major culture changes and it will be really hard. They have lost control of their process and they now have a serious problem.

Answer 6

Create & maintain RTM.

In a similar situation, I suggested & implemented a requirement traceability matrix in an automation team where requirements and test cases were mapped to find out redundant test cases and the requirement which are not covered by any of the existing test cases.

After this exercise, a surprising truth was found that 30-40% of test steps were redundant between test cases and still many requirements were only partially covered.

Answer 7

Others have had had very good points, but I have one more thing to add to managing a huge amount of tests. Make sure you have a way to take one step ( ore few steps) backwards from the test results to form groups. Generally forming groups by functionality of the software works, and you might need to add groups for end-to-end tests etc. that do not fit ti groups. Anyway, this way you can handle one group at a time instead of the whole bunch. Also, if you maintain a description of why and what you are testing, you will find it easier to keep the test integrity. If you create a tree structure out of this, you will be able to manage thousands of tests.

I am not sure how Microsoft Test Manager works this out, but I would guess it has features to support this kind of working. I am using Meliora Testlab which allows this with Jenkins-Selenium integration. This kind of approach works fine with Manual testing which is also nice.