7

My company is not an anti-automation place. In fact, one of the reasons they hired me was to bring in some expertise in test automation. The problem is that the current processes and procedures in place in the company do not currently have anything built in to allow for automation work.

The industry we service is highly regulated and so there is a need for very strict processes and procedures in order to satisfy both internal and external audits. Automation is definitely a good tool for regression testing and providing tests for data-entry intensive tasks. The problem is that the auditors need to be reassured that the automation that we are looking at creating provides the same assurance of proper functionality and security that a manual test does.

Does anyone have any experience with this sort of regulated industry and integrating automated validation and regression tests into the process? If so, I'd love to hear about how to go about it.

Edit based upon comment below: Just as clarification, the automation I'm working with is primarily automated regression and functional testing, many times operated through the UI but not limited to that. Regression and functional testing may also apply to testing of API's and WSDL's as well as testing of messaging and communication protocols, all of which are present within the regulated industry that I'm currently servicing.

Improve this question
7
  • Can you be more specific about the kind of test automation you are considering? In particular, are you looking at UI test automation or API- or protocol-level automation?
    – user246
    Commented May 12, 2011 at 15:05
  • You might want to adjust the title; it sounds as if your primary constraint isn't so much the absence of prior automation as it is needing to satisfy auditors. How about this title instead: "How do I satisfy external auditors that automated tests are at least as effective as manual tests?"
    – user246
    Commented May 12, 2011 at 15:09
  • Two your first question, a bit of both. Some of the testing involves the UI, some of it involves behind-the-scenes processes (SQL stored procs, protocol messaging, etc). Test automation, for me, kind of covers all of that stuff. As for your second comment, yeah, you're right. I've adjusted the question title appropriately.
    – TristaanOgre
    Commented May 12, 2011 at 15:34
  • I think we are getting closer to an answer. How do the external auditors determine whether manual tests are reliable, or whether manual tests are actually performed?
    – user246
    Commented May 12, 2011 at 18:52
  • Now that is a question that I'm not sure I can answer as I'm relatively new to the company and am not sure how decisions are made, at least the first part of the question I cannot answer. As for the second part, there is a rather well documented set of Standard Operating Procedures (SOPs), policies, work-flows, etc., that are executed for every software change. If manual testing needs to be performed, there's an exhaustive paperwork trail as to what was done, how it was done, what were the results, and who was involved. And that paper trail is, right now, mostly manual.
    – TristaanOgre
    Commented May 12, 2011 at 18:56

9 Answers 9

Reset to default
4

I work in the financial industry. Our company undergoes audits too. The auditors want to know you have a documented process, and they want to spot-check that you actually follow the process.

It sounds to me as if you need a written test plan that describes what each of your automated tests covers. The plan needs to be detailed enough so that if the auditor asks to see the code for a specific test case, you can quickly point them to exactly that code. You also need a record of when the automated tests are executed and what they found. Of course that could be automated too, although if your company doesn't everything manually now, you could probably start by archiving your test results in a manual way (e.g. each test summary goes in an HTML file, and the files are stored in a directory.)

Improve this answer
1
  • I think a combination of this and the suggestion of detailed logging by @Carmi is going to be the way to go. Also, @Lyndon 's suggestion for a BDD I think will also play a part eventually but the logging and manual documentation will be the best short term "bang for the buck".
    – TristaanOgre
    Commented May 12, 2011 at 19:44
5

I think the answer you need might be in logging and auditing of the automation. When the automated test runs, have it log exactly what is being done, how and when. That way, any result can be replicated as soon as it occurrs.

Also, you have a record of what happened, and you can use that to make your case. "We know the automation tested features A and B with these results, as it is in the log here and here."

Then you can decide to manually audit say 20% of all tests, repeating them manually, and then manually following the log, and comparing the three results. Make an effort to audit the most problematic tests first, to help you gain confidence.

A nice secondary advantage is that when you're running regression tests, you can compare the entire run from build to build, and not just the results. This will save you loads of time in diagnosis, and also help to spot bugs in the automation itself.

Improve this answer
3

Most of the my automation work isn't with complaince-impacting impacting applications, however, it is certainly possible to make it audit friendly.

First of all, start using a BDD tool such as SpecFlow or StoryQ. Many of the scenario's will be documented this way. I'm not familiar with SpecFlow, but, with StoryQ it will make writing out what the automated test is doing much easier. Ensure that the stories/requirementms are laid out for each critical test. By critical, I mean the ones that are going to draw the most attention. This will also provide pass/fail at each step on the final report that is generated.

Although I recommend stories/requirements in each test, if you don't, having a diagram (shudder) of the methods can help. The auditor(s) may want you to walk them through the tests. If they do, the diagrams can help them focus where they want to look and make your audit experience far less painfull.

Although this probably should be more drawn it, it may help you with a starting point.

Edit based on comment from Rakesh: Both of the tools that I mentioned are for windows only. Here are some other tools that can be of use. Disclaimer - I have not used any of the below.

Ruby: Rspec

Java: JBehave, BDoc, Instinct

Improve this answer
3
  • I'll take a look at those two tools, Lyndon, as that might actually answer one of the questions that the validation manager here had for me. Do you have any experience using StoryQ with a tool like TestComplete or QTP? Will it work with those testing tools or is StoryQ more intended for an environment like Visual Studio?
    – TristaanOgre
    Commented May 12, 2011 at 15:03
  • Lyndon the tools you have mentioned, are they only windows based
    – Rakesh Prabhakaran
    Commented May 12, 2011 at 15:29
  • Unfortunately, they are both only for windows. I'll add references to tools for other languages to my answer
    – Lyndon Vrooman
    Commented May 12, 2011 at 15:38
2

Almost all manual tests that can be done on a web based application can be automated using tools like Selenium. If you can automate a few important test cases as per your application domain which will be of highest severity and priority and also if presented with a very good result generated, must probably do the trick.

Hope my answer is in the right track .

Improve this answer
3
  • That's the tack I'm currently taking, Rakesh. We're running tests manually and automated in parallel to show that the outputs of both sets of tests are the same, that the tests are finding the exact same kinds of defects, and that the automation is actually more reliable and faster than the manual processes. The only problem with this approach is that it is very resource intensive in that it sucks up a LOT of personnel resource to do both paths until such time as the auditors are satisfied.
    – TristaanOgre
    Commented May 12, 2011 at 15:58
  • Yes ie the hard side of this method Tristaan. Until your Automation suite becomes quiet stable and the auditor gets satisfied with the efficiency of your test suit with a good coverage, you may have to spend some fair enough time on both manual and automation.
    – Rakesh Prabhakaran
    Commented May 12, 2011 at 16:33
  • It's so tough. You need to justify your time spent on the test suite, and for that you need meaningful, cost effective results. But until you can invest that time, you won't get anything back from it! It's so difficult to convince management that things like this are worth it in the long run.
    – corsiKa ♦
    Commented Jun 6, 2011 at 23:18
2

Tristaan, good question.

Here's my advice.

What Not To Do

Be careful not to imply to the auditors that automated tests are as effective as manual tests in cases where they are not designed to cover as much as the manual tests would. Julian Harty made this point eloquently in one of his recent presentations. I've included two of his slides below. His full presentation (which is quite good) is available here.

enter image description here

enter image description here

What To Do

In addition to the other good advice in the other answers, keep a clear record of what the automated tests actually test and do not test (as well as what manual tests are designed to find that the automated tests could well miss).

Improve this answer
1

It would help if you were able to map automated scripts to your requirements to demonstrate coverage is the same. You already do manual testing so presumably you have something that fulfils that purpose already. This can be something as simple as a spreadsheet.

Whatever process you have in place for updating your manual regression tests, apply the same process to updating the automated scripts.

Improve this answer
0

Since you didn't mention what industry are you in we can all guess. I guess that medical devices will require you to prove that your automation is reliable and bug free by itself which leads to parallel runs tgsrvyou mentioned. If there are auditions, aren't there specifications for proving testing reliability ? For example can't you declare your automation reliable once you compared it to a manual run ?

Improve this answer
1
  • It actually is the medical industry, pharma and medical devices, but primarily in document exchange and patient records and history. Hence the heavy regulation. Additionally, while we don't specifically deal with the devices and compounds, patient safety is part of the whole process so there is a lot of necessary reliability in the applications. As for declaring the automation reliable, it's not quite that simple. Internally, proving to our internal auditors would be easy. But when the external auditors come in, they'll need something more substantial.
    – TristaanOgre
    Commented May 12, 2011 at 18:21
0

"The problem is that the auditors need to be reassured that the automation that we are looking at creating provides the same assurance of proper functionality and security that a manual test does."

For regression testing, automation is superior to manual testing. The regression tests don't make mistake like a person can. You also have a person audit each regression test (at least when they create it and hopefully once or twice more.)

For security testing a 100,000 iteration fuzz test is provably more effective than any kind of manual testing.

There is scientific research data to back these claims up. Find the science and show it to the auditors.

Improve this answer
0

Couple of useful pointers while looking at automation

  • Automation of repetitive tasks would save time (ex-Basic verification cases / Regression Cases)
  • value of Automation would be realised in successive runs. You would need to spend initial investment on building your automation
  • Automation suite can be scheduled to run on need basis, no manual intervention
  • If an application is in sustenance / maintenance phase it is a good candidate for functional/regression automation, You neeed not have a dedicated QA team. Automation would help in this case
  • Again if you are following TDD model, Building automation would be helpful when you need to retest same module daily basis. Doing manual testing for same build every day would be a challening job
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.