Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

The testing department is new at my company so I am trying to establish some test reporting policies. We are starting out by develping a suite of regression tests which will be good for the long term stability of the programs but I don't expect to catch too many bugs this way so I'm afraid the test results will be pretty boring to start with. We have an agile development process where a new version of the programs to be tested are released every two weeks. I'm concerned that e-mailing the test results will become too routine after a while and the e-mails will eventually be treated like spam and ignored. I'd like to post the testing results on an internal web server as this will provide a historical repository of what has and has not been tested in the past but I'm concerned that this will largly be ignored by stakeholders.

Are there any best practices for presenting meaningful test results to the right people at the right times and still keeping a good history of test results?

share|improve this question
    
Question for you: Are you talking about an automated test suite of regression? Or are you talking about all test results, manual and automated? –  TristaanOgre Jul 15 '11 at 18:53
    
@TristaanOgre - Ultimately the question applies to all test results. –  semaj Jul 15 '11 at 20:31
add comment

7 Answers

up vote 5 down vote accepted

I like posting results online, and then emailing stakeholders only for failures. No email means everything went smoothly. If someone isn't sure or just wants to see what was run, then the full set of results is available online.

As TristaanOgre points out in the comments, testers need more frequent communication to be sure the tests ran. I would add that the title should mention if the run failed or passed for easy sorting into folders. For testers, then, a lack of e-mail means a lack of a test run.

share|improve this answer
2  
I like for testers to receive an e-mail even for successes as at least it indicates the tests actually did run. It doesn't need to be anything more than "Test XYZ Completed". For example, if there was a power outage over night during which the tests were supposed to run, when the power comes back on line, depending upon environment, the tests may not run at all. A lack of e-mail in this situation will indicate an adverse event that needs to be investigate. –  TristaanOgre Jul 15 '11 at 18:03
1  
What if the stakeholders do not have the link to the online test results report handy? I feel it would be more convenient to have a test report containing the details along with the online test reports link. –  Aruna Jul 15 '11 at 19:06
    
@Tristaan, I was thinking from the stakeholders' point of view. I agree that more emails makes sense for testers, and will update my answer accordingly. @Aruna, what form would this test report take? If it's emailed for every test run including successes, then you run into the "I get too much email, I will just ignore this" issue again. If you are sending emails for failures only, I would definitely recommend including a link to the online test results there. –  Ethel Evans Jul 15 '11 at 19:51
1  
The subject of the test report email should highlight the key issue if there are failures. Example "P1 - Showstopper - [Date] - [Release]" . I usually draw attention with a larger audience if there are critical failures this way. Else I just send it with normal subject line "[Date] - [Release]" –  Aruna Jul 15 '11 at 22:52
    
@Aruna, that makes a lot of sense, esp. if there is a convention so "important" reports can be easily filtered. I +1'd your response, making reports more useful and more easily filtered is possibly a better way to make sure important emails aren't missed, depending on the organization. –  Ethel Evans Jul 18 '11 at 17:52
show 1 more comment

I like to take reporting back to a couple of fundamental questions.

  1. Who are you reporting to?
  2. What do they need to know?

The answer to those questions is not constant, so it doesn't make sense to me to rely only on a constant reporting style. If you're emailing the same report to the same people week after week regardless of the work being done, then yeah it's probably going to be ignored. What I suggest you do instead is arm yourself with a repertoire of test reporting styles and pull them out as appropriate.

The things that you're working on will be different. It stands to reason that your reporting should change to suit. If you have a suite of regression checks, then great. Are the results of those important to anyone outside the dev team? Who else do you report to? What are they interested in?

If you're doing standups, then you can report anything the dev group needs to hear about there (or at least tell the relevant developers that you need their time).

If you're reporting to project managers or product team or whoever, then make sure you speak with them to find out what they want to know about before you start testing. There's no use reporting a bunch of stuff they don't care about (the caveat to that being that it's information that they don't know that they need to hear).

Reporting isn't necessarily the soulless boring drudgework that you do at the end of testing. You don't have to turn in a novel each time. There are plenty of ways to report. Here are a few. Have a think about the different situations you might employ these:

  • Drive-by reporting: As you head past your audience on the way to somewhere else, give them a 30 second update on your progress
  • Mike Kelly's MCOASTER heuristic for when people stop by for a status update
  • Paired bug repros: Grab a developer and show them a bug (or several) you've found
  • 1 page / 1 slide status update - put the relevant stuff in point form on 1 page. Give it to your audience and encourage them to ask questions
  • Low-tech dashboard: A whiteboard somewhere public with information for people to see and digest at a glance - James Bach has a good version of this.
  • Test session notes & debrief notes
  • Screen captures and/or annotated images

I'm sure you can think of more.

share|improve this answer
    
"I like to take reporting back to a couple of fundamental questions. Who are you reporting to? What do they need to know?" Bingo! The keys to all communication. –  Joe Strazzere Jul 19 '11 at 20:12
add comment

It is fine to ask others for best practices, but ultimately you must tune your own practices to the needs of your organization. You have at least two audiences: testers and developers. Their interests, attention span, and tolerance of interruptions will vary. I recommend that you propose something that sounds right to you, share it with your team, and ask for their feedback.

Whatever your team decides, expect it to change over time and as the team changes. (Your remark about spam suggests you have already anticipated this.) If possible, design your process so that how you deliver test results is loosely coupled from how you store test results, because they may vary in different ways.

share|improve this answer
    
Sounds like good advice. Aside from testers and developers, I also have project managers and upper management as audiences which are different audiences still. +1 for the advice to loosely couple the delivery method from the storage method. –  semaj Jul 18 '11 at 15:15
add comment

I had good experience with dashboards. In one extreme (very simple dash) it was CruiseControl Yahoo Widget, you get a green for all-OK, red for failure and yellow for something is wrong or not running. It was visible on a screen hanging from the lab ceiling visible for everyone...

In other places we used tables, pie charts etc. to summarize the results and make them understandable in a glance.

share|improve this answer
add comment

A bug report containing the below could be meaningful:
1. Number of test cases passed, failed and pass percentage
2. Analysis and classification of failures : This is very important. A brief failure classification helps understand whether it is an automation issue, application code bug or environment issue. If a large % of test cases fail due to automation/environment issue it alerts us of the bad health of the automation suite
3. As @Tristaan pointed out having the list of bugs with the developers in loop can help to keep the DEV informed about the priority and timeline of the bug
4. Number of TCs blocked by a bug - This again can help the developer understand how critical the bug is. If a bug blocks 300 test cases, no matter even if its a trivial issue, it needs to be fixed ASAP. You can add the DEV's manager as well in these situations

Maintaining history of test results is also important. If I want to find the delta of test cases that passed in the past release and failed in the current one, the history helps. I usually organize test results release-wise in a shared folder.

share|improve this answer
add comment

Normally I would think a defect tracking system would track defects that are open, fixes & associated resolutions and could auto-generate a report to parties who want such notification ...however, since at my firm, we don't have a robust defect tracking system, one solution I have come up is (this particular example applies to installation testing) to create a .xls file logging the issues with hyperlinks to backup documentation, then upload the .xls file to SharePoint. SharePoint can be configured to generate an e-mail anytime a file is updated so the affected parties are notified of changes - if they opt to receive those alerts.

For our team, this method is working for now, and it keeps the testers out of developers and managers offices (in other words, lessens distractions) during crunch time. The .xls file keeps a status update of each defect and that really is the bottom line for a manager so they can only focus on the critical or open issues.

A separate .xls file is maintained per version release, and since each is uploaded to SharePoint, there is a history we can easily refer back to when needed.

share|improve this answer
add comment

Unless there's a bug to report, test results are meaningless to anyone outside of the testers themselves. At least, that's my experience. Why do I need to know that we found zero bugs every morning? That's not useful to me as a business analyst, developer, etc., on a daily basis. So, the fact that it is ignored by the stakeholders is actually expected and shouldn't be a concern.

That said, the history is still important and needs to be preserved somewhere, just not broadcast to everyone. If a bug is found when running a test, the developers may want to know the version/build, date, and time of the last known successful test run so they can pinpoint the change set that potentially created the bug.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.