I'm trying propose meaningful changes to measure how well the QA department in our company is doing. I've gone through similar Stackexchange pages that go over measuring a teams/individual's performance but most of them tend to move in the direct of - "metrics are bad". I agree with that approach to a certain degree, but there must be a way to measure how well QA department, the QA teams and QA Engineers are doing.
Here's what we currently measure (not a comprehensive list)
- How many functional test cases have been automated - Regression Automation coverage by different teams.
- How much increase in Automation coverage has been achieved each quarter
- Number of defects identified in production
- Releases that were done on time
The problem with the above metrics is that the numbers although look good on paper and makes the senior leadership sleep well at night, they aren't of value to me. The Functional Automation numbers are calculated by a check box field in ALM & Jira for each test case (Automated: Yes/No). So teams happily misreport to show a gradual increase.
The leadership set 80% automation goal by the end of 2018, voila - mission accomplished.
Most of the functional automated test cases are unstable plagued by monolithic architectures of the apps and services they meant to test, environment issues, lack of stability of test cases (poor coding standards, no code reviews), overburdened QA engineers with back to back projects with no room for self reflection so on and so forth.
I have an opportunity to propose meaningful changes. But I'm struggling to find the right methodologies and metrics. Using the wrong metrics can be dangerous and discourage good engineers. But the leadership wants to grade all QA Engineers, managers and teams based on some sensible metrics. Operating in a metric-free regime appears to be out of question.
Here are some possible metrics I can think of. Are there any potential pitfalls with them and how do wego about measuring them?
- Are we (QA department) doing better or worse compared to the previous quarter - I'm guessing that the number of PROD issues might serve as one of the ways
- How well is a team adhering to the QA Test Pyramid (Working towards the right mix - Unit/Component (bigger percentage of total tests) , Service, UI and Non functional tests)
- How well are the test plans designed
- How to identify and recognize individuals/teams that are doing the right thing
- How good or bad is our hiring process Are leaders doing the right thing
I must be missing a few other important areas. Please let me know what you think.