Let's say you have product or project that is coming near to a next release and that has previous versions released a while ago and that has a big suite of automated user acceptance tests that just take too long to run, from one day to two.
How often do you run these? How do you handle bugs (failing tests) in such a large suite of tests?
If they can be run unattended, run them over the weekend every week. That'll give you enough run-time, and enough break between runs to fix bugs and perhaps improve the automation.
As for analyzing the results, that's a different issue. I have found it useful to group the results into features or categories. That way, I can quickly see from the result set where the biggest problems are, and focus on that.
As for handling bugs, it depends on the quality of the result returned and on the nature of the bug. Results should include more than just pass/fail. They should include the relevant error message, test action and parameters, and version details. This is so that they are easily turned into bugs. If that's not possible, then they are relatively easily reproduced manually and logged as bugs that way.
It depends. For me, every time there is a deployment to test environment we would run the automation. What might change though is the number of test cases that we would run and the number of machines that we would use to run them.
After we go-live with 1.0, we normally hand select a number of key test cases, that give us a high-level test pass. This test pass would be to give broad coverage across the features, targeted to quickly identify the areas that might need more attention.
There is a potential of a real risk of test escapes vs execution time balance here. I would also increase coverage in areas that a) have recently been changed and b) areas that are known to be historically unstable.
Another challenge, is that if the application that is being tested changes, you really need to run all the tests to keep them in good, working order.
I'd agree with Bruce, it depends. If your requirements need these to run more often fit them in where you can. Otherwise what you may need to do, if you can afford it, is to set up an environment to constantly run them. I did this in one company where we needed to continually run these tests and did it in a Longevity environment where the tests always ran, and we just updated the version there.
I know not everyone has the capability for this, and I don't in my current company, but when you have long tests that you need to run and you have to do it somewhere it is an option. For places where I don't have newer environments I run these in off times, weekends or vacation days, or whenever. Otherwise, I select a representative subset and run those as often as I can.
As often as you can afford to. They are automated after all. If they're robust and don't give you false positives, then ideally every commit once the compile and other tests have passed, then kick them off.
Mandatory:
Optionally:
When you move to releasing weekly and daily and by the hour and minute, your whole strategy has to change.
Instead of the large volume of traditional long E2E cases that takes many days and weeks to run, you need to switch to using the Agile Testing Pyramid and the Agile Testing Quadrants to actually eliminate the large volume of long running E2E tests. Most tests should be unit level.
E2E tests should also be written before the code (BDD) and there are many practices that go with these that need to be adopted.
A coach or mentor is useful on this journey to avoid wheel re-invention, much as discovering all these truths feels good.
