Any hints on how to automatically generate documentations about automated tests?

Question

I've been in charge of a java / selenium / TestNG / Maven / Allure testing stack for a while, and it's been growing a lot in terms of coverage and features. My concern is that, while I could onboard contributors and make a lot of people code new tests or fixes, I'm struggling to convert new devs to use the tool (it's part of our CI, but not mandatory, and there's no reason for this to change). I often receive questions like "Is this covered? How many tests do you have on this feature? How long will my test run? What scope/testgroup/tag should I use to cover this user flow?".

I would like to be able to auto generate some documentation or knowledge base (like something better than brute javadoc -_-) that will help my users find answers by themselves on what the test stack can do for them. Think: a way to search among the tests without having to read java code. Maybe something built on top of the github repo?

Anyone have experience about generating test code documentation?

PS: I could do it manually, but we all know how manual documentation ends up after a few months without enough resources dedicated to it.

Answer 1

So if someone is wondering the same, I ended up with a very custom solution:

• a bash script that parses code files and extract test methods headers / meta datas (think method name, @Description, @Test etc.) into a CSV
• then the CSV is fed as a data source for a Google Data Studio graph, and it allows for filtering among tests according to the metadata, so devs can click on their team name, or service name to see what is covered.

So the pipeline is not 100% automated, but it's fine enough to refresh info once a week, it takes less than 5 minutes.

Answer 2

To generate this type of classification of the tests, you can use TestNG's groups.

@Test (groups = {"login"}, priority=1)
public void testLogin() {...}

@Test (groups = {"logout"}, priority=1)
public void testLogout() {...}

Then your developers can simply search for groups = {"logout"} or look at the reports, if your reporting tool integrates with TestNG.