I recently joined a new project, and unfortunately, it's riddled with bugs, leaving customers frustrated despite the significant budget we've allocated for testing. I suspect that our test pyramid might not be well-balanced. I know we have some tests in place, including unit tests, automated integration tests, and manual end-to-end (E2E) tests. However, I'm trying to figure out how to assess whether our test pyramid is properly structured for this project. Any advice on how to evaluate this?
2 Answers
"Properly structured" is very open to interpretation. And it depends on what version of the test pyramid you want to use as there are several interpretations of it. In general, it can be difficult to quantify in numbers something that is qualitative by nature.
If we go with the classic pyramid of "Unit, Integration, UI," then it comes down to finding out numbers. In theory, the test pyramid really points to the amount of test automation there is in a project. You can extend that to consider other forms of quantitative measures.
Here are some questions to consider to figure this out:
- How many unit tests are there? e.g. 78 unit tests
- What is the unit test code coverage? Consider different types like statement coverage, branch coverage, lines of code (loc) covered. e.g. 57% statement coverage, 25% branch coverage, 45% loc
- How many integration tests are there? (And by integration, do you mean API level or another form of integration?) e.g. 25 integration tests, 0% automated.
- How many UI tests are there? e.g. 200 manual tests, 0% automated.
And if there is more than one code repository, you'll need to run the numbers on each one to get an accurate assessment.
Now, if you are writing a report for your team/leadership on the current state of testing and test automation, I'd extend this to include some other metrics. These will help you determine if testing has been a priority. Not all of these may be relevant, just some ideas to consider. Maybe this will help you think of more areas to assess and quantify.
- What's the pass/fail rate of any automated tests? If they fail often, are they catching bugs or are they false positives/negatives?
- How many open bugs are in the backlog? e.g. 150 open bugs
- How old is the oldest bug in the backlog? e.g. 3 years 5 months or # of days.
- Average age of open bugs in the backlog? e.g. 1.7 years
- How many bugs are fixed/closed per sprint?
If you're using Jira, you'll likely have to use a custom JQL command to find these numbers.
"leaving customers frustrated"
- Is there a way to quantify this at all? This one may be more difficult to gather information about. You may need to talk to customer service managers/leads to help get this info.
- Can you correlate the number of open customer support tickets there are?
- Number of calls to the call center?
- If customers are canceling service, how many over what period of time?
- Can you find out if the number of user complaints go up/down after a software release to production?
-
Two minor nit-picks: a) How many bugs are fixed/closed per sprint? -> Not all development is done in sprint iterations. b) How many open bugs are in the backlog? e.g. 150 open bugs -> A number of bugs does not convey the state of an application. One would preferably categorize bugs by impact and time needed to fix them if presenting this to stakeholders.– PromeCommented Aug 20 at 13:41
-
@Prome these are just example questions and considerations to take into account when assessing quality metrics. As I put in the disclaimer, "Not all of these may be relevant, just some ideas to consider." I would hope people would see that and take what works for them and leave the rest. Or, apply these to their specific situation, e.g. not using sprints...fine, just replace sprints with some other measure of time. These considerations are highly adaptable and that's how they should be treated. Commented Aug 20 at 18:15
Usually I use data when I need to assess the quality of something.
In your case I would dedicate a period of time, collect all the bugs and perform as many RCA (Root Cause Analysis) as possible with a small twist- the end result should be how could we detect this bug using testing, and where is the ideal point to do that. A side effect could be improving quality in general, but it's not related to this question.
I added "ideal point" because you'll need to take into account the bigger picture, and not only one bug. For example bug X could be detected using unit-tests, but combined with bug Y and Z integration testing could be a better solution.
Choose a period that can provide enough data but not be too long, a release or two can be a good choice.
As a side note, sometimes is easier to use the Swiss Cheese model (see the image below taken from wikimedia) when thinking about stopping bugs from reaching production.
