How to assess Regression Risk without looking at the source code?

Question

A new practice was introduced during our last retro:

• In each ticket, QAs should provide a brief assessment towards Regression Risk if this new feature / fix is implemented.

My question is:

• It is hard to assess Regression Risk without looking at the source code; our sprint is three-week long, even we could look at the source code, there is not enough time to do so. Are there any good rules of thumbs when providing Regression Risk assessment without looking at its source code?

Answer 1

First, I would suggest that estimating and understanding risk should be a shared activity (include developers, PMs, other stakeholders). While QA often understands a fair bit about some of the risk associated with software work, "scheduling risk" and other concerns that may not fall on QA should usually also be considered. Bringing in more roles will often also help to uncover additional risks that a single role may not think about or provide context around the impact if it occurs. When performing this exercise collaboratively or reviewing other roles' feedback it can be an opportunity for everyone to gain a deeper more complete understanding of areas of risk in the system you're all working in.

For the question you actually asked, I believe you're talking about what I've usually been exposed to as "Risk Analysis". When I've been involved in risk analysis we approach each proposed change to the system from the angle of...

1. "What might go wrong" (direct and indirect to the change if its integrated with other modules, etc.)
2. "How likely do we think these things are to go wrong" (this can be a gut feeling or backed by something more factual)
3. "If those things do go wrong, what's the probable impact to customers of the software" (how many people will it effect, what will happen to them, for how long, etc.)

This can be more of a loose-feel conversation or you can attempt to quantify using estimates in a number range. Once you have an idea of what might go wrong and what's the biggest deal if it breaks, you can use this assessment to plan the scope of your regression testing to target areas of greatest risk.

Other than my answer, I'd highly recommend searching the internet for "Risk Based Testing". I think you'll find a lot more/better information than I can personally provide.

Answer 2

First - couple of my thoughts about regression risk assessment

• Any brief assessment implies pretty low level of accuracy
• Personally I do not see much value in assessing regression risk briefly. What is that?
  • Is it a probability of regression issue? If yes, how do you use that information? Say we estimate 15% probability of we will get regression here, so what (we don't know what sort of regression it will be, brief assessment won't give us any details)?.
  • If you need to estimate the time to address regression phase of your testing, I would concentrate straight on the "time" measure, but not on the "risk".

Now, answering your question

I would suggest statistical method. To have that method working more or less accurate you should maintain as much fields of your issues as possible, like:

• Component
• Who developed the change (which particular developers)
• What is the volume of the change (lines of code, number of files, etc)
• What time did it take to implement the change
• How many issues are related to that change
• etc.

So having this historical data for all the changes you have tracked, you can statistically estimate what would be the risk of regression of your new change.