Selenium test coverage visualization

Question

We have a huge legacy application where unit/integration tests are impractical and most automation is done in Selenium with Java programming language in a separate automation project.

There are about 1000 @Tests written, the tests use Page Object Factory implementation and they check:

• business rules (requirements)
• critical user journeys
• UI interactions

When looked at individually it is rather clear what each test does. What we are missing is having a clear high level picture what test coverage do we actually have.

Our goals are:

• visualize what we have covered
• finding out what tests are we missing
• identifying possible duplicate tests

Is it possible to get some kind of visualization automatically generated?

Otherwise how to even get a handle on test coverage of a complex business rule heavy application without terrible manual grunt work. However if manual approach is the only solution, in what way should we even present all that data for it to be useful?

Answer 1

I feel your pain. I live your pain: I am in the same situation, except I'm the only tester in the group and it's taken me years to get any stable, working test automation running.

I'm not aware of any tooling that can be used to measure coverage in this kind of situation. When you're dealing with large legacy software where unit tests and integration tests aren't practical, you don't have a tool-friendly way to report your coverage.

That said, I do have a suggestion:

Build your coverage reporting into your automation

You won't be able to say that you cover x% of the code branches in the application. What you can do is take a little time to create - or acquire from help documentation or some other source - a high level overview of the application functions, and map your existing and new tests to application functionality list.

For instance, if your software is a web store you might list functions like "add products to cart", "clear cart", "browse products", "search products", "create customer account", "log in", "log out", "purchase", and so on. These would become categories that your allocate to your tests, possibly grouped into larger categories like "products", "cart", "accounts". Depending on your application, the categorization could go multiple levels deep - but you would start at the highest level and get more detailed over time.

Once you've got your tests categorized (I would consider starting with the very basic "categorized" and "uncategorized" so the process can be something that happens whenever you are working on the test code - in essence, if you aim to categorize say 5 tests any time you edit or add a new test, you will get the whole thing done eventually), you can work on extending your reporting to include the categories.

Ultimately, you would want to finish with a chart of some description that shows the number of tests in each category and how many of those passed with each test run.

By building this into your automation and using the test result reporting to generate the chart, you'll have a large (and probably slow and painful) start to the reporting, but it will be maintainable and extensible - any time new features are added, you add a new category that starts with 0 tests and grows.

This won't have the same level of accuracy or detail that coverage tools for unit/integration tests provide, but it will give you an idea of which areas of the application have less coverage than others.