6

We have a quality processes department who are responsible for processes in our organization to meet FDA standards. It seems though that they require more of us than is actually verified by FDA auditors. Unfortunately this department has a lot of political power so we pretty much have to do anything that they tell us.

Anyway, one of the things they require us to do is to record a lot of stuff during the execution of tests. So for example if I test the performance of a calculator app, instead of having a test that looks like this:

Action: Multiple 2342352345 by 131345356452 and measure the time it takes to perform the calculation Expected result: It took no more than 2 minutes

The test would look like this:

Action: Multiple 2342352345 by 131345356452 and record the time it takes to perform the calculation Expected result: Recorded time = ?

Action: Verify (Pass/Fail).............................................................................................................. Expected result: It took no more than 2 minutes

Action: Record a screenshot of the result and attach it to the test............................................. Expected result: Screenshot was recorded and attached.

Can you explain why they require us to do that?

Improve this question
2
  • 2
    For those outside the United States, the FDA is the Federal Drug Agency.
    – user246
    Commented Jun 3, 2014 at 12:22
  • 3
    The FDA is the Food and Drug Administration. Your link is correct, but the label is not.
    – John Oglesby
    Commented Jun 6, 2014 at 3:20

5 Answers 5

Reset to default
9

This is not necessarily the case, but I would suspect that your processes department has been bitten in the past by documentation not meeting FDA standards or by FDA auditors whose expectations are higher than the FDA standards. So they've effectively adopted a CYA approach ("cover your anatomy") and insist that absolutely everything that can be documented gets documented.

That said, they do have valid reasons for their approach.

The sample format you've supplied requires explicit documentation of the actual output. For the kinds of software that are subject to FDA standards and FDA auditing, that data could mean the difference between winning and losing an expensive lawsuit (or having people die because of a software glitch...). Whether your software is life-critical or not isn't relevant - it's still being held to that level of standard.

The screenshot requirement is added evidence that the test was actually performed as specified. I know that sounds bad, but when faced with a mountain of repetitive manual tests, it's perfectly normal and human to not perform them precisely as written (particularly since in my experience the software updates faster than the tests can be updated to match so half the time it's impossible to perform the tests as written anyway).

The short version is that your processes department is requiring both the tests and the evidence that the tests were performed so that they have the ability to prove to FDA auditors that everything is meeting standards whenever an issue arises.

Improve this answer
5
  • So you are saying the primary reason for this (except pacifying overzealous FDA auditors) is to make sure testers actually perform the tests rather than simply click on the "Pass" button? But why would testers mark pass if they didn't actually do the step? If they cheat, they can cheat by providing an unrelated screenshot as well. Isn't that so?
    – user3251930
    Commented Jun 3, 2014 at 11:30
  • 3
    Of course they can cheat that way - but that takes extra effort and in my experience there's never enough time. My experience is that a tester starts tracing something through because something doesn't look right, then goes back to the test cases and can be inclined to say "Oh, yes, I did that while I was doing this other thing and it was okay." Most of the time that's not a problem - but sometimes it can be.
    – Kate Paulk
    Commented Jun 3, 2014 at 11:36
  • 1
    Very good answer and was exactly as I thought when I read the question. Lawsuit, fines and possibly loss of contracts/ability to distribute the product versus implementing "CYA" methodology ...... CYA always wins. Plus at the end of the day it will likely result in a couple more defects being detected since if properly monitored all of the test cases are being correctly executed and thoroughly tested throughout the system.
    – Paul Muir
    Commented Jun 5, 2014 at 13:38
  • This is very probably the actual reason for this requirement. However, this approach could actually increase lawsuits. Specifying documentation of a specific result would require it be tied to a specific test case. That approach tends to inhibit exploratory testing, which could uncover many additional flaws that users will eventually find. Whether those static test cases exhibit "due diligence" would be something lawyers would love to argue over.
    – Jeff_Lucas
    Commented Jun 10, 2014 at 14:28
  • @Jeff_Lucas - oh, without a doubt. The lawyers would win, too. Lightweight tools to log actions during an exploratory session help, but they're not really enough to satisfy that particular brand of risk-averse that tends to see anything can't be directly traced and linked to a cost/benefit as unnecessary.
    – Kate Paulk
    Commented Jun 10, 2014 at 17:38
5
+25

There is a better level of precision in your second example. A simple pass/fail evaluation gives you one piece of data - that the calculation ran in less than 2 minutes. But if this same test is run 6 months from now and still comes in under 2 minutes, will you know whether performance has increased, decreased, or remained the same?

Just this week I tested an optimization to a query. The Acceptance Criteria was only that the query run in less than 30 minutes (it is a large data load). For my test environment, I ran the test before applying the changes, and it ran in 31 seconds (a Pass). After the optimization, I ran the query against the same data, and it took 53 seconds (also a Pass). In both the Before and After tests, the Acceptance Criteria of < 30 Minutes was met, but it was worth another look at the query to see why it took longer after the fix than it did before. A simple Pass/Fail evaluation would not have initiated that second look.

As to the screenshot, as Kate has said, that is just a way to guarantee that the work was done when the tester said it was done. Screen shots typically have time and date from the system clock. If there are ever concerns, the time of code changes can be compared to test time to guarantee the correct code was tested, or to explain discrepancies (if the test result was obtained prior to the code being moved into QA, for example).

Improve this answer
3

For the FDA you are required to report on traceability from each product Requirement to the passing Test Result. Information captured needs to include details of the test environment, acceptance criteria, objective evidence demonstrating that the requirement has been implemented and statement whether the test passed or failed.

In your test case example above, the expected result "Screenshot was recorded and attached." is not really needed. This is not a verification of a product requirement. This is a process step. Write your test cases so that the only objective evidence which you gather is directly in support of verification of requirements (i.e., there is no product requirement about screenshots being recorded and attached).

Hopefully, you don't have to deal with physical print-outs of objective evidence which need to be initialed and dated.

BTW: if you are ever invited to appear in front of an auditor, you will be grateful that there is this level of documentation about what was done. It isn't always the manager who is questioned.

Improve this answer
2
  • Thank you. Is a "pass" result not sufficient for the auditors? Do they also want to see recordings of data or other evidence? Don't they trust the testers to pass only those steps that met the requirements?
    – user3251930
    Commented Jun 8, 2014 at 8:36
  • Our quality management system requires objective evidence. Objective evidence has been examined during FDA audit.
    – Jean Marchant
    Commented Jun 13, 2014 at 13:51
1

FDA standards are not my area of expertise, but I will give this a shot based on working in similar systems (ISO 9001, electromagnetic compatibility testing).

First, your example is a bit strange in that your expected results should be in the form of some limit or threshold. For example, the first example above should be:

Action: Multiply 2342352345 by 131345356452 and measure the time it takes to perform the calculation Expected result: Action complete in less than 2 minutes Actual result: It took no more than 2 minutes

With that, the requirement to take a screenshot of the result would make sense if the screen displayed the time in seconds. However, to record it properly there should be two screenshots: the first to record the intial time and action initial state and a second to record the time and showing the action completion.

With that preface, here are the most likely reasons for the more elaborate procedure:

  1. People change jobs. The evidence you create during the tests have to stand alone without having to interview the person conducting the test. Just having a tester check a box indicating "Pass" would not be sufficient documentation of the test.

  2. Memory is unreliable. Even if the person is still at the job, multiple experiments have shown that memories can be faulty even a short time after an event (e.g. the gorilla in the room). The test results should be sufficient to indicate the completion of the test, independent of the tester's memory of the event.

  3. Procedures can be wrong or incomplete. In your example, the "intent" may have been to hand-calculate the numbers and enter the result back into the computer. If the second screenshot showed the result in the calculator app in Windows, that would alert the reviewer of a process problem.

  4. The test results should be independent of tester skill or organization culture. Hand-written results require good hand writing. Typed results require a good grasp of corporate terminology. A screenshot is relatively independent of these factors.

Hope that helps!

Improve this answer
4
  • Thank you. However, you answered the question assuming that documentation of tests execution is required. But let's go deeper. Why is it required? If there is some problem in the field, then you can try just install the released version and try to reproduce the problem. You don't need to look at the test execution documentation. Why would you ever need to look there? Another question, did you have a policy that required recording details about the tests execution in the companies you worked at?
    – user3251930
    Commented Jun 5, 2014 at 17:38
  • True, you can always rerun the test. However, you did specify that this is for (I assume) FDA certification. Let's say you ran the test and it passed. One year later, you find that the software failed in the field and the failure is traced to that test scenario. You set up the system to rerun the test and ... the test fails. That is the reason for the documentation. This is not a case of CYA, but a necessary step for tracing root cause of the problem.
    – Jeff_Lucas
    Commented Jun 6, 2014 at 15:08
  • So are you documenting test execution in your company?
    – user3251930
    Commented Jun 8, 2014 at 8:37
  • Absolutely not! Again, the question was why someone would require that level of documentation (i.e. the thought process behind it). I have had teams that were working on very critical processes that did require that level of documentation, but that was long ago. Personally, I think it is a complete waste of time. It is better to characterized "how" the tests are performed instead of detailed results (unless there is a bug). BTW, I have seen that level of detail a few years back using the QTP GUI automation tool. It was never of much use to me.
    – Jeff_Lucas
    Commented Jun 9, 2014 at 23:45
0

I work in financial services in the UK as a business analyst. I am currently on a project that has a second "big" consultancy to do the work.

The first one was kicked out, they were photoshopping the test evidence (screenshots) and hiding the defects (yes it was one of the big 4) the system had so many defects the programme had to go into a "stabilisation" programme to sort out the mess. The second "big" consultancy has a contractual obligation to allow the business analysts access to the system and (read-only) access to the test database.

The second consultancy was also providing test evidence with screens shots with no commentary. I have found the test evidence showing the code defects where the coder has changed the format of the field content (Was: yyyy-mm-dd To: yy-mm-dd oo:oo:oo:oooo) and he thought it was ok. The tester just did a screenshot and passed it. I have rejected this and provided a test evidence template.

The template requires the user story details, the acceptance criteria, above each test case they need text to explain what the test is about, the before and after versions of only the screens that have changed. An extract of the relevant test record from the database.

I don't trust what they deliver because of what I have found, not once but many times. So documented test evidence is critical to proving a system.

Template layout: This provides a good set of test evidence - accurate, only what is needed and complete. This enables you to trace the test back to the requirements.

  • Test to be performed
  • Testers name & date of test
  • User story
  • Acceptance criteria
  • Text explaining what is specifically being tested
  • Before: Screenshot of screen being changed
  • After: Screenshot of screen being changed
  • Before: DB extract of the relevant data record
  • After: DB extract of the relevant data record
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.