Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Our organisation and clients expects us to give a sort of test "certificate" for each release.

Each release consists of a couple of three/four weeks sprints, including testing efforts as:

  • Automated unit-tests
  • Automated integration-tests
  • Exploratory testing sessions (during the sprint)
  • Manual end-to-end test (each sprint)
  • Manual regression suites (each release)

What kind of release test report do you deliver to an enterprise client keeping in mind Agile principles like:

Working software over comprehensive documentation

I have looked at "traditional" testing reports and they include documenting any findings during testing. Also I have seen test reports including screen-shots to prove the testing happened. This all feels like a great waste of time, because there is a very big chance no-one will read them ever.

So my question is "What do we (minimally) report to give the client the feeling we do thoroughly testing and satisfy critical managers who expect test reports?"

share|improve this question
    
Can you ask the client and managers what they want to see and why? Can you start a conversation with them? –  Phil Kirkham Jul 16 at 11:40
    
Not with the client, and the manager thinks we (the test team) should come up with something solid :) In our industry materials are always delivered with a test certificate, management thinks we should do something similar. But my question is broader I think, Agile teams often clash with tradinational thinking management. How do you get the best of both world with the least effort. –  Niels van Reijmersdal Jul 16 at 11:43

4 Answers 4

I have served as QA lead for multiple Scrum teams. Any release might have multiple teams releasing at the same time. I was responsible for the overall QA sign-off and test summary.

What we provided as a summary document:

  • What user stories were included from each group
  • Any documented defects that were not fixed and therefore constituted a release risk
  • Highlights of specific test efforts - these were not detailed, more on the order of "Executed standard automated regression suites on components X, Y, Z" or "Built and executed new performance test suite for component A"

We didn't typically provide a list of defects other than those that were still open and constituted a release risk or a problem for our customer service desk, nor did we provide a list of test cases executed. If there were concerns about a particularly tricky change or new component, we might provide links to our RST documents to internal partners.

share|improve this answer

Having done 23 years of performance tests including trouble shooting, correction, and rerun I can agree with testerab on long reports full of nice graphs that remained unread (and even if scanned certainly not understood). Of course if you are a paid for third party, or even expendable employee, that report might be the only way you can justify your costs so you have an interest to make it fat.

To some degree of course your reporting depends on the findings.

If the testing does uncover issues, these are resolved (even if just workarounds to allow test completion), and then reruns show success the justification for a fat report to ensure findings are understood and can be implemented is fair enough.

But what happens when the test simply works? Tons of test planning, building, script engineering, sighter tests, formal tests, result validation - and the user says, well what a waste of time, always knew it would work. Maybe a fat report could demonstrate your efforts?

But ignoring justification I'd suggest:

  1. short high level easily understood Management summary (include 'thank you's to any user who gave input: discussed SLA, demo'ed how they used it, participated in 'look and feel' testing, etc.)
  2. link to test plan. Document "changes from plan" only if need to justify extra time or cost
  3. brief precise numbered conclusions
  4. brief precise numbered recommendations
  5. invitation to discuss

and leave all the data available for browser viewing in a general repository.

8 pages at most including cover and content.

HTH

share|improve this answer

I do not know what documentation you currently has or what tools you use. So I do not know is all of the following possible for you. But maybe you found some helpful ideas.

Defect Overview

  • How many defects was created.
  • How many defects was fixed/closed.
  • How many defects of the ceated/fixed/closed has a low, medium or high priority. (Maybe you has more types of priorities.)
  • How many defects was found in what kind of testing phase.
  • If you have different sections/components in your software, how many defects was found in what kind of section.

Test Cases Overview

  • How many test cases executed.
  • How many test cases failed/passed at the end of the testing phase.
  • Maybe you should also show what priority the failed test cases has.
  • If you have different kind of test sets for the different sections of the software split the report in these. So the people can see what section is the most critical.

Requirements

  • A simple list who shows if a requirement covered by a test case or not.

This all should be simple generated by a tool like HPQC. Most of the data can be shown as graphs, to it is easy to read and not so hard to understand.

share|improve this answer

This all feels like a great waste of time, because there is a very big chance no-one will read them ever.

I hear you on this one. I have wasted far too many hours of my life producing documents that nobody ever read - enough so that at one point, I devised Anna's Chocolate Bar Test. (Somewhere in the last half of the document, you state that you have chocolate in your desk for the determined few who have read this far. Then you see who comes to you for treats. Obviously this is not one to use on client documentation!).

However.

The agile manifesto also states "while there is value in the items on the right, we value the items on the left more". That does not mean that documentation is something you should avoid, only that you should not allow creation of documentation to take the place of creating working software. Sometimes documentation is actually a crucial part of the deliverables for your product (in a regulated industry, for example), and that's fine. There is absolutely no contradiction between a good Agile process and creating necessary documentation.

In this case - I can see your difficulty because you are not (for some reason) being allowed to discuss the client's needs with the client, so it will not be tailored to their needs. But you still have to produce something that they will like and that conveys all the work you have done.

I've a couple of suggestions for reading:

1) Read up about low tech testing dashboards - this is something I have not found the need to use much myself (yet), but I think it's a good match for an Agile process, especially on an ongoing project which lasts through a number of sprints. Here's how Marlena Compton implemented this in an Agile environment at Atlassian.

2) I think it's also worth reading Michael Bolton's posts about telling the testing story.

For myself, the reporting I do on a release basis is aimed at a slightly different goal: I want teams to have conversations, to understand if anything they are releasing might combine in a bad way with something another team is doing. I want our release managers, should there be an issue, to be able to make decisions, and know who to go to for more information. What we produce is a two page document - a table summarising the main areas affected by this release, which team worked on it, what percentage of our customers use that functionality, and some comments on what we think the main risks are, and what we've done to investigate them. This is useful both for the teams themselves, and the release managers. (I'm also sure it will change plenty over time - this is a snapshot!)

I don't think my current approach would necessarily work well in other environments, as it depends very much on a culture of transparency. Sometimes companies have a culture where they pretend that risks they don't talk about don't exist, so you shouldn't talk about a risk unless you have mitigated it. That's I think why traditional test reporting often seeks to obfuscate risks in an ink-cloud of dodgy metrics. Only you can tell whether either your company, or your client, might have that culture.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.