I have heard a lot of "abuse" about automated testing, especially UI bases automated tests. As UI is very fragile and prone to change (especially in agile shops). Personally my automated tests have given more confidence about working system than finding defects.
What is your success ratio with automated UI tests? Or maybe how you define success with automated UI test?
My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs.
But what they do do is two key things:
They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs.
They give you the opportunity to heavily re-factor or make large changes quickly and confidently.
The previous posters are correct - automation can never find new bugs, that can only be done by sapient testing - however automation will find regression faults, and often not the ones you are looking for. However, sapient analysis of the results of automation tests can find significant issues.
I cite an example for my last project where we had close to 1000 automated tests running through the UI every day. We had an occasion where some of the tests were failing intermittently and without any clear reason (the code around that area was stable and hadn't been touched for some time).
So when I looked at the test results, I couldn't see anything immediately. One of the test criteria was that the page title was "foo", and when I looked at where the assert had failed I could see that the actual page title contained "http 403".
Just by looking at that failed assertion, I immediately had a rough timescale for the failure, and could more quickly isolate the completely unrelated problem that was unintentionally bringing down the site for a few hours perhaps once a week.
I can't say if our test team would have found that bug or not without the regression suite, but it would have taken much longer for sure.
You have to understand the following...
Automation will never find NEW bugs.
It will find bugs if your system is updated. It will find bugs if the system is down for whatever reason and you are sleeping in your bed and you have automation enabled overnight. It will send you an urgent email that you will read in the morning that the site was down for an hour or so.
It won’t find bugs that humans find through bug regression. It won’t find bugs in the system that require analysis and constant pounding. It won’t find hidden bugs. It will only find bugs that it is programmed to find.
That said... Automation is something that will help you concentrate on those hidden bugs. It will get a LOT of testing off your shoulder. Like multy browser, endless input of various combinations of data. Tidies tasks like create test data, companies, users, etc. Stress test, Load test, functionality. All that can be covered.
So success for an automated test would be the assurance that if a developer checks in code, you can be sure that a major part of the application is regressed at night when you are sound asleep. And you can be sure that it will find changes, defects which that update produced and nothing will be unnoticed unless you didn't write a test for it.
I find that automated tests in general can be very useful for ensuring that older functionality isn't getting broken by new code, and I have seen new bugs in new code uncovered by old regressions tests. I try to test as much functionality as I can without the UI, and then do some additional integration tests to make sure the UI is calling the right functionality after the functionality itself has been verified. This practice results in far more maintainable tests.
I prefer to avoid UI testing whenever it is not necessary to get the information I want from that test due to the maintainability costs. In my experience, UI automation does less to reduce technical debt compared to other forms of automation because of this ongoing maintenance debt. I prefer to write more maintainable functional tests first, and then test the UI manually if I run out of time (all else being equal, that is). Of course, some projects have a higher UI test maintenance debt than others, and automation efforts should be prioritized accordingly. The two main sources of maintenance issues in the projects I worked on were UI changes and timing changes. UI tests tend to timeout if they don't see expected behavior, so one failed test doesn't block execution of the remaining tests, but there is often noise that can also lead to timeouts and false failures.
When I do finally get to the point where UI testing is the highest priority automated testing remaining, I find that some issues with UI testing can be greatly mitigated by good design. All timeout values and CSS / XPath / accessibility navigation / etc. strings should be in one place, where they can be easily configured. Although I hadn't heard the name, I've learned with trial and error to use the "Page Object" technique, and it helped a lot with our UI test design. Good communication with developers is also a must, to avoid investing a lot of time writing tests that will immediately be broken due to a UI change. Developers can also often design their code to be as testable as possible by other automated tests, so you can minimize the surface area you are targeting with UI tests to just examining the GUI logic and the interaction with the rest of the code, and not functionality testing.
One thing to consider is that in many cases your regression suites are working off of a static set of regression data or criteria. One way that regression suites can be of higher value is trying to develop heuristic oracles vs. static ones. In a talk I went to at Seaspin on 3/1 Harry Robinson talked about moving from the static test automation to a more dynamic variety. In one slide he gave an example:
Heuristic Test Oracles
- Verify some results for the SUT
Check other results using
– simple algorithms
– consistency checksStatic oracle: sqrt(17) = 4.1231056
Heuristic oracle: sqrt(N) * sqrt(N) = N
From his presentations slides which are available on the seaspin site above. The idea behide this being if you can test a much broder series of values in this sort of manner.
Yes, our automated test suite can and does find critical bugs. Not very often but just off the top of my head I can think of two occasions when it did.
I agree with the answers above that the main benefit of automated tests is freeing up your human testers from performing repetitive tasks and so allowing them to concentrate on the really intelligent testing they're good at. I would add though that the regression tests we automated were among the most important areas of the application (most frequently used by our end users) so when the automated tests did catch bugs they were really, really important ones.
As a developer, automated regression tests have certainly saved my day several times. But a QA professional may not be aware of this. Here is the reason:
A deterministic test can find a bug only once: When I (the developer) run the test suite and see that I made a stupid mistake, I fix this bug immediately --- and no tester is going to see it. Without the test suite, the bug would make it into the repository.
From my experience, I most benefited from simple integration tests: When an apparently benefitial change in one component had an undesired effect in another component.
