1

We have a huge legacy application where unit/integration tests are impractical and most automation is done in Selenium with Java programming language in a separate automation project.

There are about 1000 @Tests written, the tests use Page Object Factory implementation and they check:

  • business rules (requirements)
  • critical user journeys
  • UI interactions

When looked at individually it is rather clear what each test does. What we are missing is having a clear high level picture what test coverage do we actually have.

Our goals are:

  • visualize what we have covered
  • finding out what tests are we missing
  • identifying possible duplicate tests

Is it possible to get some kind of visualization automatically generated?

Otherwise how to even get a handle on test coverage of a complex business rule heavy application without terrible manual grunt work. However if manual approach is the only solution, in what way should we even present all that data for it to be useful?

  • Firstly, you need define what type of coverage are you looking for. This answer shows it is not a simple question: sqa.stackexchange.com/a/44995/12740. Secondly, what's the status of instrumentalization of the SUT? You could use the logs that your application creates to investigate the actions provoked by your test code. – João Farias Oct 28 '20 at 11:24
  • The questions are a bit different, at this point we are not trying to measure our coverage but to visualize what our tests really cover. When we know what the tests cover, we can aim to increase that test coverage in any aspect mentioned on that thread (requirement coverage, user roles coverage, timings/interuptions/concurrency coverage etc.). We have a pretty good idea what needs to be tested in general, but it's hard to have a true grasp on contents of such a high number of automated tests and the knowledge of what have we actually covered so far. Some logging does exist. – fing Oct 28 '20 at 13:42
2

I feel your pain. I live your pain: I am in the same situation, except I'm the only tester in the group and it's taken me years to get any stable, working test automation running.

I'm not aware of any tooling that can be used to measure coverage in this kind of situation. When you're dealing with large legacy software where unit tests and integration tests aren't practical, you don't have a tool-friendly way to report your coverage.

That said, I do have a suggestion:

Build your coverage reporting into your automation

You won't be able to say that you cover x% of the code branches in the application. What you can do is take a little time to create - or acquire from help documentation or some other source - a high level overview of the application functions, and map your existing and new tests to application functionality list.

For instance, if your software is a web store you might list functions like "add products to cart", "clear cart", "browse products", "search products", "create customer account", "log in", "log out", "purchase", and so on. These would become categories that your allocate to your tests, possibly grouped into larger categories like "products", "cart", "accounts". Depending on your application, the categorization could go multiple levels deep - but you would start at the highest level and get more detailed over time.

Once you've got your tests categorized (I would consider starting with the very basic "categorized" and "uncategorized" so the process can be something that happens whenever you are working on the test code - in essence, if you aim to categorize say 5 tests any time you edit or add a new test, you will get the whole thing done eventually), you can work on extending your reporting to include the categories.

Ultimately, you would want to finish with a chart of some description that shows the number of tests in each category and how many of those passed with each test run.

By building this into your automation and using the test result reporting to generate the chart, you'll have a large (and probably slow and painful) start to the reporting, but it will be maintainable and extensible - any time new features are added, you add a new category that starts with 0 tests and grows.

This won't have the same level of accuracy or detail that coverage tools for unit/integration tests provide, but it will give you an idea of which areas of the application have less coverage than others.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.