How do I effectively measure the level of impact of test automation within a Safe Scrum environment?

Question

Details:

Currently we are within a Safe Scrum environment and have already automated much of manual testing. However, we would like to measure the effectiveness and coverage of test automation across teams.

To do this, we have been working on overview pages via Jira + Confluence + X-Ray.

However, we still have some problems to directly measure the result from the other teams, more than 40 teams in total (no kidding!) and to create a good overview from it.

Do you have any advice on what could be improved in such an environment?

Initial situation

• Safe-Scrum environment with 40 teams and different projects.
• We have all teams and its coverage covered via the respective Jenkins pipelines and here via X-Ray, Jira (Testcases + Cucumber).
• However, measuring the true effectiveness of teams cannot really be standardized.

Questions:

• How can I unify the information from the different test teams within a Safe Scrum environment?

Answer 1

To bring unity and coherence to the different test teams in a Safe Scrum setting, consider the following approach:

1. Test automation coverage: This metric indicates the extent to which the application is being tested using automated tests. For example, if an application has 100 test cases, and 80 of them are automated, the test automation coverage would be 80%. Tools like X-Ray, Jira, and Jenkins can be used to monitor this metric and ensure that the coverage remains adequate as the application evolves over time.

2. Test automation efficiency: This metric measures how much time it takes to run automated tests compared to manual tests. For example, if manual tests take 8 hours to complete and automated tests take 4 hours to complete, the test automation efficiency would be 50%. Jenkins can be used to track this metric and optimize the automation process to make it more efficient over time.

3. Defect detection rate: This metric assesses how many defects are being detected through automated tests compared to manual tests. For example, if manual tests detect 20 defects and automated tests detect 30 defects, the defect detection rate would be 150%. Tools like X-Ray and Jira can be used to monitor this metric and ensure that automated tests are finding as many defects as possible.

4. Release readiness: This metric reflects the confidence level of the development team in releasing the application to end-users. This can be determined through regular surveys of the development team, asking them to rate their confidence level on a scale of 1 to 10. For example, if the development team rates their confidence as a 9 out of 10, it would indicate that they are highly confident in the release readiness of the application.

5. User satisfaction: This metric measures the satisfaction level of end-users with the application. This can be determined through regular surveys of end-users, asking them to rate their satisfaction level on a scale of 1 to 10. For example, if end-users rate their satisfaction as an 8 out of 10, it would indicate that they are generally happy with the application.

6. Root cause analysis: This technique involves identifying the source of defects and testing issues in order to improve the test automation process. For example, if automated tests are consistently failing for a particular feature, root cause analysis can be used to identify the reason for the failures and find ways to improve the tests. Tools like X-Ray and Jira can be used to track this information and help the development team make data-driven decisions about improving the test automation process.

7. Test process improvement: This technique involves continually enhancing the test automation process through regular reviews and retrospectives. For example, after every sprint, the development team could conduct a retrospective to identify areas for improvement in the test automation process. Jira can be used to track this information and help the team prioritize and track progress on process improvements over time.

Answer 2

Start a community of practise, which is a standard practise of SaFE:

Communities of Practice (CoPs) are organized groups of people who have a common interest in a specific technical or business domain. They collaborate regularly to share information, improve their skills, and actively work on advancing the general knowledge of the domain.

https://www.scaledagileframework.com/communities-of-practice/

Let the CoP experiment, and slowly figure out what can be standardized, and what should be, so that it values the people using it.

The CoP should be able to understand your companies context, take suggestions, design what should be uniform and feel the pain of their decisions