Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

We currently use the following processes for testing:

  1. Write a test plan (currently stored in Microsoft Test Manger)
  2. Execute Test plan
  3. Once the tested functionality is promoted to Production, automate the Test plan using Borland SilkTest, for future regression testing

Question : Is this a typical process for a shop that uses automation? As changes to the original test plan change (due to future modifications), do you continue to maintain both the manual test plan as well as the automated one?

Thanks!

share|improve this question
    
What does the test plan include? What level of abstraction? –  dzieciou Mar 23 at 14:04
    
The plans are pretty specific (enter "John" into input marked "first name") –  Rudesyle Mar 24 at 15:19
add comment

3 Answers 3

up vote 2 down vote accepted

It is probably different for each organization/team/product/process, but here is a typical process I have used (starting from new functionality):

  1. Identify the key test parameters such as oracles, surfaces (variables like platform, inputs, outputs, etc), and risk areas.

  2. Explore the function under scrutiny at the time using multiple manual test sessions.

  3. Develop automation to make testing that function faster or identify functions that would make a good addition to a smoke test (fully automated) suite.

  4. Integrate automation into regression test automation suite as smoke test or independent computer assisted test modules that help setup or check specific areas of the application. This step is done as a part of the weekly or biweekly cycle while it is fresh.

  5. Only document enough to identify which automation to use with each regression test area. The function of the scripts are documented as comments in the script itself.

The key here is that regression automation is going to break on a regular basis, so a smart mixture of automated and manual test for regression is best to make it as useful as possible but still maintainable. The scope of the automation will work itself out over time based on the time you have available for implementation and maintenance.

Hope that helps!

share|improve this answer
    
Everyone's answers have been great. Once you automate a manual test session, do you continue to maintain it over time, even after it's been automated? –  Rudesyle Mar 24 at 15:22
    
Additional difference I can see between Jeff's process and Rudesyle's one is that Jeff's does not define test processes upfront, but do it in more explorative way. –  dzieciou Mar 24 at 15:23
    
The point about exploratory approach is well taken. I have also been involved in waterfall government projects that required validation of static requirements in a fixed time frame. That project more closely reflects the process in the description. The intent there, however, was to demonstrate functionality and not a good process for finding defects. –  Jeff_Lucas Mar 24 at 15:41
    
The topic of automated script maintenance is important. When the script breaks, you have pressure to quickly get it working again, but you need to understand whether the failure is caused by an intended change. Naturally, that requires the automation be maintained "real time". The person running the scripts should be both an experienced tester and an experienced script writer to be efficient - a tough combo to find. –  Jeff_Lucas Mar 24 at 15:46
add comment

As the other answers have said, there is no "typical" process. Everyone handles things their own way.

That said, there are some common factors:

  • most places will have some form of test plan repository. I've seen everything from a mix of documents and spreadsheets in a shared directory structure to the high dollar professional tools.
  • most places will figure the test plan gets executed manually. It's rarely a good idea to automate manual test plans - what constitutes a good manual test is not necessarily a good candidate for automation.
  • most places that automate will have a way to identify tests targeted for automation and a strategy to add targeted tests to their regression codebase. The exact strategy varies a lot.
  • places that are serious about automation will not use record/playback whether their tool supports it or not.
  • places that are serious about automation will treat their automation codebase as code
  • places that are serious about automation will keep their automation codebase in a version control repository. It may not match the application code version control repository, but if it isn't matching the application codebase version control, it will be flagged in a way that allows regression to be run against any version of the application codebase required.
  • places that are serious about automation will ensure that time is devoted to maintaining automated regression.
  • automated regression for a new feature will not be built until that feature is stable.

It sounds from what you've said that you've got most of these points covered, so you're probably on the right track with your process. The main thing is that it works for your situation.

share|improve this answer
    
I'm curious to know why you feel that serious Automation shops avoid the recording mechanism? We use it to capture a test, and then tweak with custom scripts –  Rudesyle Mar 24 at 15:20
    
@Rudesyle - There's nothing bad about using a recording mechanism to capture the identifiers for a test. If you don't customize and parameterize (and reuse that code), you end up with an unmaintainable mess that's loaded with repeated and near-repeated code. In an app where you've got to log on to do anything, a standard record/playback test will log on, do something, log off. Imagine having to update thousands of those because of a change to the logon screen and you see the problem. –  Kate Paulk Mar 28 at 11:00
add comment

Firstly, each company has its own test processes but it is typical in common. Secondly, each company has different development lifecycles so testing processes do differ. I can share with you our current test process:

  1. Overview project structure
  2. Overview diagram of requirements
  3. Get the software requirements specification document
  4. Write a test plan which includes the following sections: purpose and scope, functions to be tested, functions not to be tested, test approaches, test pass/fail criteria, regression test strategy, tools (test case management, bug tracking, performance testing tool, automation testing framework etc), test schedule, risk management
  5. Prepare the smoke test
  6. Create a functional test suite based on the software test plan which was created in the previous step
  7. Prepare the automation test suite
  8. Then agile process (verifying features in the end of each iterations), develop and updating automated regression test suite
share|improve this answer
    
In step 5 you mention prepare a smoke test. Isn't a smoke test just a quick test where you click on some links to ensure nothing on the screen "blows up" ? A test plan, to me, would detail pass/fail scenarios –  Rudesyle Mar 24 at 15:24
    
@Rudesyle You are right actually but we have separated checklist for smoketest in our process. But, again, I have described our process :) Smoke testing is usually quick check of basic functionality and does not need separated test suite. –  amazpyel Mar 24 at 15:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.