Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Let's say you have product or project that is coming near to a next release and that has previous versions released a while ago and that has a big suite of automated user acceptance tests that just take too long to run, from one day to two.

How often do you run these? How do you handle bugs (failing tests) in such a large suite of tests?

share|improve this question
1  
Automated user acceptance tests? I don't follow. Is it a person familiar with how your users think doing it? That'd be a user acceptance test. Or is it a script doing it? That'd be an automated test. Unless you can automate your actual users' responses to your application, I'm not sure you can do automated user acceptance tests. I might be missing something fundamental here... –  corsiKa May 9 '11 at 5:39
    
I'm referring to the automated UI tests that are as faithful as possible to the client/user's requirements. –  Edgar Gonzalez May 9 '11 at 5:46
3  
So automated functional testing then. :-) –  corsiKa May 9 '11 at 7:00
2  
@glowcoder "acceptance tests" is a clunky description, but it is actually a common description. (I don't like the terminology much myself, as neither word is actually true, but it's what people seem to use). –  testerab May 9 '11 at 21:15
add comment

5 Answers

up vote 10 down vote accepted

If they can be run unattended, run them over the weekend every week. That'll give you enough run-time, and enough break between runs to fix bugs and perhaps improve the automation.

As for analyzing the results, that's a different issue. I have found it useful to group the results into features or categories. That way, I can quickly see from the result set where the biggest problems are, and focus on that.

As for handling bugs, it depends on the quality of the result returned and on the nature of the bug. Results should include more than just pass/fail. They should include the relevant error message, test action and parameters and version details. This is so that they are easily turned into bugs. If that's not possible, then they are relatively easily reproduced manually, and logged as bugs that way.

share|improve this answer
add comment

As often as you can afford to. They are automated after all. If they're robust and don't give you false positives, then ideally every commit once the compile and other tests have passed, then kick them off.

share|improve this answer
add comment

Mandatory: 1. Every release (to the solution team) 2. Every change on a daily load base that is core or affects more than one already delivered functional areas 3. Huge number of bug fixes.

Optionally: 1. Daily smoke/acceptance tests

share|improve this answer
add comment

It depends. For me, every time there is a deployment to test environment we would run the automation. What might change though is the number of test cases that we would run and the number of machines that we would use to run them.

After we go-live with 1.0, we normally hand select a number of key test cases, that give us a high level test pass. This test pass would be to give broad coverage across the features, targeted to quickly identify the areas that might need more attention.

There is potential of a real risk of test escapes vs exectuion time balance here. I would also increase coverage in areas that a) have reciently be changed and b) areas that are known to be historically unstable.

Another challenge, is that if the application that is being tested changes, you really need to run all the tests to keep them in good, working order.

share|improve this answer
add comment

I'd agree with Bruce, it depends. If your requirements need these to run more often fit them in where you can. Otherwise what you may need to do, if you can afford it, is to set up an environment to constantly run them. I did this in one company where we needed to continually run these tests and did it in a Longevity environment where the tests always ran, and we just updated the version there.

I know not everyone has the capability for this, and I don't in my current company, but when you have long tests that you need to run and you have to do it somewhere it is an option. For places where I don't have newer environments I run these in off times, weekends or vacation days, or whenever. Otherwise I select a representative subset and run those as often as I can.

share|improve this answer
2  
I think this is a good compromise: choose a representative sample of tests, and run them during otherwise dead times: weekends or overnight. Perhaps run the full suite over the weekend, and the representative sample overnight. –  Peter K. May 9 '11 at 12:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.