Let's say you have product or project that is coming near to a next release and that has previous versions released a while ago and that has a big suite of automated user acceptance tests that just take too long to run, from one day to two.

How often do you run these? How do you handle bugs (failing tests) in such a large suite of tests?

  • 1
    Automated user acceptance tests? I don't follow. Is it a person familiar with how your users think doing it? That'd be a user acceptance test. Or is it a script doing it? That'd be an automated test. Unless you can automate your actual users' responses to your application, I'm not sure you can do automated user acceptance tests. I might be missing something fundamental here...
    – corsiKa
    May 9, 2011 at 5:39
  • I'm referring to the automated UI tests that are as faithful as possible to the client/user's requirements. May 9, 2011 at 5:46
  • 3
    So automated functional testing then. :-)
    – corsiKa
    May 9, 2011 at 7:00
  • 2
    @glowcoder "acceptance tests" is a clunky description, but it is actually a common description. (I don't like the terminology much myself, as neither word is actually true, but it's what people seem to use).
    – testerab
    May 9, 2011 at 21:15

6 Answers 6


If they can be run unattended, run them over the weekend every week. That'll give you enough run-time, and enough break between runs to fix bugs and perhaps improve the automation.

As for analyzing the results, that's a different issue. I have found it useful to group the results into features or categories. That way, I can quickly see from the result set where the biggest problems are, and focus on that.

As for handling bugs, it depends on the quality of the result returned and on the nature of the bug. Results should include more than just pass/fail. They should include the relevant error message, test action and parameters, and version details. This is so that they are easily turned into bugs. If that's not possible, then they are relatively easily reproduced manually and logged as bugs that way.

  • Note that in 2011 releases were monthly or yearly. Now in 2020, daily or hourly or minutely is very common. If that is the case for you 'over the weekend' is no longer a viable or useful strategy. Mar 4, 2020 at 13:19

It depends. For me, every time there is a deployment to test environment we would run the automation. What might change though is the number of test cases that we would run and the number of machines that we would use to run them.

After we go-live with 1.0, we normally hand select a number of key test cases, that give us a high-level test pass. This test pass would be to give broad coverage across the features, targeted to quickly identify the areas that might need more attention.

There is a potential of a real risk of test escapes vs execution time balance here. I would also increase coverage in areas that a) have recently been changed and b) areas that are known to be historically unstable.

Another challenge, is that if the application that is being tested changes, you really need to run all the tests to keep them in good, working order.


I'd agree with Bruce, it depends. If your requirements need these to run more often fit them in where you can. Otherwise what you may need to do, if you can afford it, is to set up an environment to constantly run them. I did this in one company where we needed to continually run these tests and did it in a Longevity environment where the tests always ran, and we just updated the version there.

I know not everyone has the capability for this, and I don't in my current company, but when you have long tests that you need to run and you have to do it somewhere it is an option. For places where I don't have newer environments I run these in off times, weekends or vacation days, or whenever. Otherwise, I select a representative subset and run those as often as I can.

  • 2
    I think this is a good compromise: choose a representative sample of tests, and run them during otherwise dead times: weekends or overnight. Perhaps run the full suite over the weekend, and the representative sample overnight.
    – Peter K.
    May 9, 2011 at 12:37

As often as you can afford to. They are automated after all. If they're robust and don't give you false positives, then ideally every commit once the compile and other tests have passed, then kick them off.

  • That must be a pure hell to kick off tests that take 2 days to finish after every commit. That was the original question anyway. I suppose you might want to change your strategy after reading the initial question thoroughly.
    – pavelsaman
    Mar 4, 2020 at 18:51


  1. Every release (to the solution team)
  2. Every change on a daily load base that is core or affects more than one already delivered functional areas
  3. Huge number of bug fixes.


  1. Daily smoke/acceptance tests

When you move to releasing weekly and daily and by the hour and minute, your whole strategy has to change.

Instead of the large volume of traditional long E2E cases that takes many days and weeks to run, you need to switch to using the Agile Testing Pyramid and the Agile Testing Quadrants to actually eliminate the large volume of long running E2E tests. Most tests should be unit level.

E2E tests should also be written before the code (BDD) and there are many practices that go with these that need to be adopted.

A coach or mentor is useful on this journey to avoid wheel re-invention, much as discovering all these truths feels good.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.