What is the best way to report test results?

Question

The testing department is new at my company so I am trying to establish some test reporting policies. We are starting out by develping a suite of regression tests which will be good for the long term stability of the programs but I don't expect to catch too many bugs this way so I'm afraid the test results will be pretty boring to start with. We have an agile development process where a new version of the programs to be tested are released every two weeks. I'm concerned that e-mailing the test results will become too routine after a while and the e-mails will eventually be treated like spam and ignored. I'd like to post the testing results on an internal web server as this will provide a historical repository of what has and has not been tested in the past but I'm concerned that this will largly be ignored by stakeholders.

Are there any best practices for presenting meaningful test results to the right people at the right times and still keeping a good history of test results?

Answer 1

I like posting results online, and then emailing stakeholders only for failures. No email means everything went smoothly. If someone isn't sure or just wants to see what was run, then the full set of results is available online.

As TristaanOgre points out in the comments, testers need more frequent communication to be sure the tests ran. I would add that the title should mention if the run failed or passed for easy sorting into folders. For testers, then, a lack of e-mail means a lack of a test run.

Answer 2

Unless there's a bug to report, test results are meaningless to anyone outside of the testers themselves. At least, that's my experience. Why do I need to know that we found zero bugs every morning? That's not useful to me as a business analyst, developer, etc., on a daily basis. So, the fact that it is ignored by the stakeholders is actually expected and shouldn't be a concern.

That said, the history is still important and needs to be preserved somewhere, just not broadcast to everyone. If a bug is found when running a test, the developers may want to know the version/build, date, and time of the last known successful test run so they can pinpoint the change set that potentially created the bug.

Answer 3

A bug report containing the below could be meaningful:
1. Number of test cases passed, failed and pass percentage
2. Analysis and classification of failures : This is very important. A brief failure classification helps understand whether it is an automation issue, application code bug or environment issue. If a large % of test cases fail due to automation/environment issue it alerts us of the bad health of the automation suite
3. As @Tristaan pointed out having the list of bugs with the developers in loop can help to keep the DEV informed about the priority and timeline of the bug
4. Number of TCs blocked by a bug - This again can help the developer understand how critical the bug is. If a bug blocks 300 test cases, no matter even if its a trivial issue, it needs to be fixed ASAP. You can add the DEV's manager as well in these situations

Maintaining history of test results is also important. If I want to find the delta of test cases that passed in the past release and failed in the current one, the history helps. I usually organize test results release-wise in a shared folder.

Answer 4

I like to take reporting back to a couple of fundamental questions.

1. Who are you reporting to?
2. What do they need to know?

The answer to those questions is not constant, so it doesn't make sense to me to rely only on a constant reporting style. If you're emailing the same report to the same people week after week regardless of the work being done, then yeah it's probably going to be ignored. What I suggest you do instead is arm yourself with a repertoire of test reporting styles and pull them out as appropriate.

The things that you're working on will be different. It stands to reason that your reporting should change to suit. If you have a suite of regression checks, then great. Are the results of those important to anyone outside the dev team? Who else do you report to? What are they interested in?

If you're doing standups, then you can report anything the dev group needs to hear about there (or at least tell the relevant developers that you need their time).

If you're reporting to project managers or product team or whoever, then make sure you speak with them to find out what they want to know about before you start testing. There's no use reporting a bunch of stuff they don't care about (the caveat to that being that it's information that they don't know that they need to hear).

Reporting isn't necessarily the soulless boring drudgework that you do at the end of testing. You don't have to turn in a novel each time. There are plenty of ways to report. Here are a few. Have a think about the different situations you might employ these:

• Drive-by reporting: As you head past your audience on the way to somewhere else, give them a 30 second update on your progress
• Mike Kelly's MCOASTER heuristic for when people stop by for a status update
• Paired bug repros: Grab a developer and show them a bug (or several) you've found
• 1 page / 1 slide status update - put the relevant stuff in point form on 1 page. Give it to your audience and encourage them to ask questions
• Low-tech dashboard: A whiteboard somewhere public with information for people to see and digest at a glance - James Bach has a good version of this.
• Test session notes & debrief notes
• Screen captures and/or annotated images

I'm sure you can think of more.

Answer 5

It is fine to ask others for best practices, but ultimately you must tune your own practices to the needs of your organization. You have at least two audiences: testers and developers. Their interests, attention span, and tolerance of interruptions will vary. I recommend that you propose something that sounds right to you, share it with your team, and ask for their feedback.

Whatever your team decides, expect it to change over time and as the team changes. (Your remark about spam suggests you have already anticipated this.) If possible, design your process so that how you deliver test results is loosely coupled from how you store test results, because they may vary in different ways.

Answer 6

Normally I would think a defect tracking system would track defects that are open, fixes & associated resolutions and could auto-generate a report to parties who want such notification ...however, since at my firm, we don't have a robust defect tracking system, one solution I have come up is (this particular example applies to installation testing) to create a .xls file logging the issues with hyperlinks to backup documentation, then upload the .xls file to SharePoint. SharePoint can be configured to generate an e-mail anytime a file is updated so the affected parties are notified of changes - if they opt to receive those alerts.

For our team, this method is working for now, and it keeps the testers out of developers and managers offices (in other words, lessens distractions) during crunch time. The .xls file keeps a status update of each defect and that really is the bottom line for a manager so they can only focus on the critical or open issues.

A separate .xls file is maintained per version release, and since each is uploaded to SharePoint, there is a history we can easily refer back to when needed.

Answer 7

I had good experience with dashboards. In one extreme (very simple dash) it was CruiseControl Yahoo Widget, you get a green for all-OK, red for failure and yellow for something is wrong or not running. It was visible on a screen hanging from the lab ceiling visible for everyone...

In other places we used tables, pie charts etc. to summarize the results and make them understandable in a glance.

Answer 8

I do not believe there are well defined best practices for this space. Your concerns about emails are probably valid but if your team can try to maintain as close to a 100% pass rate then any dip should be noticed, it's only when there are constantly failing tests that no one is fixing that everything starts to be ignored. I think just like many development practices no matter how you report results this needs a single owner at the top, if you have a daily stand up then if there are any failing tests the lead should assign each of the fails to someone to fix, the next day ensure that they were fixed.

To help fix issues quickly appropriate logs, artifacts, etc. should be logged and easily available. In truth reporting is treated like an afterthought everywhere I've worked but it is a mistake. After all the time spent writing automated tests or performing manual tests the reporting is the only way other developers and managers actually see the return on the whole thing. Even if that means all of your tests are passing all the time that tells you something, that no one is breaking anything, and if the tests are all passing but things are actually breaking then that tells you the testing is not adequate. Yes the results may be 'boring' as you say if they rarely fail but they can also be useful for understanding the test coverage you have.

If no matter what you do everyone at the company ignores the test results, especially leaders, you have a cultural problem, that's far harder to solve than anything else, people want automation and more software but often don't want to invest in what 'watches the automation'. This needs to be solved because otherwise a lack of robustness will eventually come to ahead and then it's the poor people at the front line under stress (developers and testers) when really it was the leadership not paying attention and failing to ask, what's our health like? How are we testing? Results reporting makes that clear so you are right to think about it.

I am affiliated with an automated test results reporting provider called Tesults so beware of my bias but if you don't have anything set up to report results currently and want to focus on actually writing tests instead of writing reporting infrastructure it may interest you. You run your tests however you want with whatever framework and at the end of the test run have a script to upload your results, it stores all test results history, files, emails results out if you want it, and allows you to triage failing tests to team members. It is a commercial cloud based solution but I know they are currently exploring how to best provide service to open source projects with public views so if you are working on something like that the home page currently says to reach out.