1

Historically at our company we have taken screenshots of every step in a test script, even if no bug has been found.

We are now moving from a waterfall method to iterative development cycles.

Over the last 6 months, we have been trialing using rapid test design using mindmaps and not taking screenshots, and instead stating that the test cases in the mindmaps and the management spreadsheets are evidence that the test has been thought of and executed.

It's our belief that taking screenshots does not prove anything and actually wastes valuable testing time. However, there is some resistance with people believing the mind map or similar processes do not give them sufficient evidence if questioned at a later date about that test.

I would be interesting in getting advice or others' experiences specifically relating to the following questions. Either way, we do not plan on purchasing recording tools.

  1. Do you take screenshots to prove a script has been executed or just when a bug is found?
  2. Have you found not taking screenshots to evidence a test, has caused any confrontation or issues?
  3. Have your found any disadvantages using rapid test case design and mindmaps for test analysis?

Please can you state whether you work for a transnational, national, regional, or startup company. We are small, national company and would like to hear from everyone, but need take a balanced view as to what we can achieve and not achieve.

  • In your experience have these screenshots ever been useful in a dispute? How much time is spent taking them? – Phil Kirkham Jul 8 '14 at 16:42
  • What industry do you work in - for safety critical and some other domains there could be a need for recorded evidence – Phil Kirkham Jul 8 '14 at 16:47
3

Anecdote: I worked on one project for a national company. Testing was being done and overseen by one of the Big Name consultancies. Their testers were told to take screenshots for every test step. They abandoned this practice as (1) it was taking too much time and test progress was too slow and (2) I was fnding 90%of the defects on the project by doing exploratory testing.

1

In my experience, the requirement to collect evidence (like screenshots) of tests being executed is driven by regulations in your industry. I had to collect evidence while testing military avionics (MIL-STD-2167), telecommunications equipment (TL9000), and financial reporting for publicly traded company (Sarbanes-Oxley).

When working with software that is not regulated, the evidence collection was not required, and generally not done. (I can't remember anytime when we did it)

You may want to check if your are required to collect that evidence before dropping it. Otherwise, if you are not required - then its a business decision on your part. Is it worth the time to collect detailed results of tests that pass? You would know by how many times you actually used the data that you collect. In my experience, its better to focus on collecting data/screenshots for failed tests to help diagnose the failures quicker.

Good luck.

  • This is more or less what I was going to say. My company operates in a regulated environment (healthcare/pharmaceuticals) and so we have to abide by ICH guidance and follow GxP. As such, all our manual test execution must be peer reviewed and so the test results must be recorded in a way that the reviewer can compare the acceptance criteria against the evidence. "Evidence" doesn't mean screenshots, it can just be a testers written observation. However, the observation cannot simply be that the step passes, so in reality a screenshot showing the results is often more efficient. – rpcutts Jul 22 '16 at 16:16
0

We record a video of our test after design of what a "good" test case pass looks like and put it in shared storage. After that videos are only kept on fail. This way we can always go back and look at what the original good run looks like.

0

To document testing scenarios, we use a wiki with text-based steps. Our wiki (FOSWiki) has version control, so you can even see how steps changed over time (and who made changes). Way better than docs on shared drive - hyperlinks make navigation a breeze. And FOSWiki allows to attach images to pages. Linking to other wiki pages with related/similar scenarios and/or bug to bugzilla is trivial. We also use (highly customized) bugzilla (which also allows attaching images) but we use it only if it helps to clear something - in 95% cases, text is enough.

We are international company with agile process and high-availability environment (100% Python) and have web-based app.

Id we started from scratch, we would go with TRAC, which integrates webSVN, wiki, bug database and has markup in code to create hyperlinks to wiki and bug database when code changes are viewed in browser via webSVN. It is pretty slick, free/open source, but alas we are too deep in our bugzilla and wiki. But we lustily look at TRAC integration and considering converting anyway.

http://trac.edgewall.org/

0

I am working for a top large enterprise software company. I've worked for internet company and consumer product companies. Assume you mean taking screenshot for manual test execution. I never heard that screenshot is taken even no bug is found. If the intention is to simply prove the test is executed, you team got a trust and management issue, address it first.

0

Everything we do as testers requires some trade off between time (planning and executing) and achieving our testing mission. Taking screen shots to reference later or to have proof of coverage is fine as long as you balance that with time to achieve your mission (including adequately testing the system). Realistically this means if we want to record our steps we must have some way to automatically take the screen shots through a tool or just use video.

If you are using Windows (7 and later I believe) you can open up Problem Steps Recorder, hit record and it will do all of this for you. I'm sure there are other tools out there that will accomplish the same things.

I'll use tools like PSR while doing exploratory testing so I can review what I've done and even perhaps find a few interesting things that I missed while my attention was somewhere else. This often leads to follow up testing.

Note: Recording every step for review later is not the same thing as having to upload or place each photo into a tracking device. Doing this would probably defeat the efficiency gains in automatically recording the steps.

Note 2: Outside of the original tester these recordings don't mean much anyways. It's the testers job to report on what they found, to summarize and provide useful information back to the product team.

0

Other comments have covered the screenshot taking, so I address the third questions: Encountered disadvantages in using mindmaps for test analysis.

So: Sometimes projects are long and complex, and on those the state of testing is hard or next to impossible to interpret from mindmaps. Still, capturing screenshots is overkill in most of the situations. The best alternative in many situations is somewhere in between. A bit boringly it is nothing new - just documenting what is relevant to test in reusable format, tracking the outcome of testing and maintaining this information so that it is easy to find out what is best to test next. And that can be done with well ordered test cases. For test cases do consider what is relevant to write down. Is it fully scripted instructions, or is it possible just write down instructions what to test, not how to test. That is often faster and thus tields better bang for testing time.

Test cases are not always not a bliss. It is possible to go wrong with them in many ways, but in complex, long projects the "old method" provides good tools to work efficiently and maintain understanding what is going on in the project.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.