4

Our organisation and clients expects us to give a sort of test "certificate" for each release.

Each release consists of a couple of three/four weeks sprints, including testing efforts as:

  • Automated unit-tests
  • Automated integration-tests
  • Exploratory testing sessions (during the sprint)
  • Manual end-to-end test (each sprint)
  • Manual regression suites (each release)

What kind of release test report do you deliver to an enterprise client keeping in mind Agile principles like:

Working software over comprehensive documentation

I have looked at "traditional" testing reports and they include documenting any findings during testing. Also I have seen test reports including screen-shots to prove the testing happened. This all feels like a great waste of time, because there is a very big chance no-one will read them ever.

So my question is "What do we (minimally) report to give the client the feeling we do thoroughly testing and satisfy critical managers who expect test reports?"

2
  • 1
    Can you ask the client and managers what they want to see and why? Can you start a conversation with them? Jul 16, 2014 at 11:40
  • Not with the client, and the manager thinks we (the test team) should come up with something solid :) In our industry materials are always delivered with a test certificate, management thinks we should do something similar. But my question is broader I think, Agile teams often clash with tradinational thinking management. How do you get the best of both world with the least effort. Jul 16, 2014 at 11:43

5 Answers 5

4

Having done 23 years of performance tests including trouble shooting, correction, and rerun I can agree with testerab on long reports full of nice graphs that remained unread (and even if scanned certainly not understood). Of course if you are a paid for third party, or even expendable employee, that report might be the only way you can justify your costs so you have an interest to make it fat.

To some degree of course your reporting depends on the findings.

If the testing does uncover issues, these are resolved (even if just workarounds to allow test completion), and then reruns show success the justification for a fat report to ensure findings are understood and can be implemented is fair enough.

But what happens when the test simply works? Tons of test planning, building, script engineering, sighter tests, formal tests, result validation - and the user says, well what a waste of time, always knew it would work. Maybe a fat report could demonstrate your efforts?

But ignoring justification I'd suggest:

  1. short high level easily understood Management summary (include 'thank you's to any user who gave input: discussed SLA, demo'ed how they used it, participated in 'look and feel' testing, etc.)
  2. link to test plan. Document "changes from plan" only if need to justify extra time or cost
  3. brief precise numbered conclusions
  4. brief precise numbered recommendations
  5. invitation to discuss

and leave all the data available for browser viewing in a general repository.

8 pages at most including cover and content.

HTH

3

This all feels like a great waste of time, because there is a very big chance no-one will read them ever.

I hear you on this one. I have wasted far too many hours of my life producing documents that nobody ever read - enough so that at one point, I devised Anna's Chocolate Bar Test. (Somewhere in the last half of the document, you state that you have chocolate in your desk for the determined few who have read this far. Then you see who comes to you for treats. Obviously this is not one to use on client documentation!).

However.

The agile manifesto also states "while there is value in the items on the right, we value the items on the left more". That does not mean that documentation is something you should avoid, only that you should not allow creation of documentation to take the place of creating working software. Sometimes documentation is actually a crucial part of the deliverables for your product (in a regulated industry, for example), and that's fine. There is absolutely no contradiction between a good Agile process and creating necessary documentation.

In this case - I can see your difficulty because you are not (for some reason) being allowed to discuss the client's needs with the client, so it will not be tailored to their needs. But you still have to produce something that they will like and that conveys all the work you have done.

I've a couple of suggestions for reading:

1) Read up about low tech testing dashboards - this is something I have not found the need to use much myself (yet), but I think it's a good match for an Agile process, especially on an ongoing project which lasts through a number of sprints. Here's how Marlena Compton implemented this in an Agile environment at Atlassian.

2) I think it's also worth reading Michael Bolton's posts about telling the testing story.

For myself, the reporting I do on a release basis is aimed at a slightly different goal: I want teams to have conversations, to understand if anything they are releasing might combine in a bad way with something another team is doing. I want our release managers, should there be an issue, to be able to make decisions, and know who to go to for more information. What we produce is a two page document - a table summarising the main areas affected by this release, which team worked on it, what percentage of our customers use that functionality, and some comments on what we think the main risks are, and what we've done to investigate them. This is useful both for the teams themselves, and the release managers. (I'm also sure it will change plenty over time - this is a snapshot!)

I don't think my current approach would necessarily work well in other environments, as it depends very much on a culture of transparency. Sometimes companies have a culture where they pretend that risks they don't talk about don't exist, so you shouldn't talk about a risk unless you have mitigated it. That's I think why traditional test reporting often seeks to obfuscate risks in an ink-cloud of dodgy metrics. Only you can tell whether either your company, or your client, might have that culture.

3

I have served as QA lead for multiple Scrum teams. Any release might have multiple teams releasing at the same time. I was responsible for the overall QA sign-off and test summary.

What we provided as a summary document:

  • What user stories were included from each group
  • Any documented defects that were not fixed and therefore constituted a release risk
  • Highlights of specific test efforts - these were not detailed, more on the order of "Executed standard automated regression suites on components X, Y, Z" or "Built and executed new performance test suite for component A"

We didn't typically provide a list of defects other than those that were still open and constituted a release risk or a problem for our customer service desk, nor did we provide a list of test cases executed. If there were concerns about a particularly tricky change or new component, we might provide links to our RST documents to internal partners.

1

I do not know what documentation you currently has or what tools you use. So I do not know is all of the following possible for you. But maybe you found some helpful ideas.

Defect Overview

  • How many defects was created.
  • How many defects was fixed/closed.
  • How many defects of the ceated/fixed/closed has a low, medium or high priority. (Maybe you has more types of priorities.)
  • How many defects was found in what kind of testing phase.
  • If you have different sections/components in your software, how many defects was found in what kind of section.

Test Cases Overview

  • How many test cases executed.
  • How many test cases failed/passed at the end of the testing phase.
  • Maybe you should also show what priority the failed test cases has.
  • If you have different kind of test sets for the different sections of the software split the report in these. So the people can see what section is the most critical.

Requirements

  • A simple list who shows if a requirement covered by a test case or not.

This all should be simple generated by a tool like HPQC. Most of the data can be shown as graphs, to it is easy to read and not so hard to understand.

0

We do require to provide a TER(Test Exit Report after every Master Build Regression testing which is bundle of all the fixes.

The contents of TER is

  • Tests Executed (Which are Automated, Manually checked, Exploratory sessions) Fixes going out in this release - Impact on Areas of product Tests excluded (reasons) Bugs found Status of those bugs Test Duration Testers Involved, Total Effort Any Risks/Recommendations Metrics (Diagrams of % tested, Automated, excluded...)

Global Dev and technology team will consider this TER as a Sign off from Tester's team to start with User Acceptance Testing.Product Owners across the globe like in 15 different countries will read this TER may be not completely but they will make sure to check Recommendations/Risks content which will guide them to test anything in particular.

As there are about 15 different global teams involved in Master build release I think it's OK to provide a 2-3 page document at the end of testing phase.

We do provide Test Summary or Test Exit reports for smaller projects which involves 1 day - 1 week testing via email no specific document will be sent in such cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.