Showing system test coverage for customers/stakeholders in agile

Question

I work for a software consultancy as a test analyst and was looking for some advice on testing in agile.

Since my company moved agile, I think that testing has turned into mini iterations of waterfall, rather than true agile.

example:

• 2 week sprint starts
• all backlog tickets are scripted
• wait for requirements/functionality to be developed
• functionality is released to test environment
• test scripts executed against SUT
• Rework/test reporting until sprint is complete

The issue I have with this approach is that we are never given the time to fully test everything.

I want to move to a more agile way of testing (e.g. light weight scripts, exploratory testing, mind map test planning for mapping out requirements), but I feel that I would struggle to extract any metrics from this approach.

For testers working in an agile environment, how to you provide metrics to customers? I usually give a breakdown of requirement coverage via test suites then give metrics from test case results to show a percentage of requirements tested, how many have passed/failed.

Are test cases necessary to show what you will test, or is the time better spent analyzing the actual system?

Answer 1

In my opinion, metrics around test cases and requirements are not the way to go for Agile testing. It is quite easy to add a lot of simple test cases that do not add that much value, but which could alter the metrics positively.

I personally find the following metrics most useful in Agile testing:

Root Cause Analysis

Root Cause Analysis shows what caused the bugs that are still in the system and why they have escaped testing.

Personally, I also prefer into dividing the escaped bugs into whether it would have added enough value versus costs if they would have been found earlier. A bug that only happens at one customer because their setup if very specific could have only been found earlier if you test all oddities of all setups of all customers. That is not worth it due to its costs versus the benefits (i.e. the bugs that would be uncovered due to extra testing). This gives two categories:

• Bugs that should have been found earlier
• Bugs that should not have been found earlier

After you have done a Root Cause Analysis into those two categories, you can show whether the test process located more bugs with the highest priority versus fewer bugs with a lower priority (rather find a functional problem than a textual inconsistency). Do note that the cause of bugs not being found early enough is not always a blame to the test team, f.e. it could be a requirements-issue (customer expected something else than is delivered).

This shows that you perform Agile testing as expected: assure the quality with most certainty in least amount of time (as time is almost always the limiting factor).

Amount of open bugs

You can also keep a graph of the open bugs at any moment in time. Ideally you measure this into priority categories. Critical bugs that need fixing within days are more important than textual inconsistencies. Note that it is not the main priority of the test team to keep the amount of bugs as low as possible, but it is a team-effort (developers need to fix them in the end).

Bugs per category

You could also keep track of the amount of bugs per category, where you can define categories as you see most fit. You could use categories like performance, security, functionality, etc., or based on components of your product (for Stack Overflow: questions & answers, search, profiles, etc.). This gives a good insight into what categories need more attention, from either testing (if a lot of security bugs arise, you should make time early so that less and less escape the test process) or from developing (to fix the bugs).

Answer 2

I'd disagree; metrics and test plans are important. (I think there's a separate question in there about how test should work in an agile environment; I'd recommend watching http://lisacrispin.com/2017/06/02/agile-testing-essentials-livelessons-video-course/ if you haven't before, and possibly asking a separate question.)

However, I don't think they're important from a client point of view. Either you're done with a function and ready to deliver it, or you aren't. It's most likely meaningless to tell them that 70% of test cases are passing, or something, because you'd then have to explain which ones aren't passing, and if they need to worry, and so on.

As for the question about some sort of documentation for both planning your test and tracking it, both are fundamental. Not because you're necessarily going to show it to anyone external, but because you are going to need to share it with your developers and other testers when you do iteration planning/standups. And you can talk to them about what the best way to document/measure things is for everyone involved. Having a hierarchy of tests can be helpful, so you can quickly show the impact; that not being able to do test A means you can do these other four tests either, whereas not being able to do test B is only blocking that specific test.

Test plans can also be important for historical purposes; it's very helpful for me as a tester to look at what someone did a few years ago when testing some similar functionality to give me ideas for what I might want to do. It can also be helpful when doing regression/other sorts of testing.