Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I have heard a lot of "abuse" about automated testing, especially UI bases automated tests. As UI is very fragile and prone to change (especially in agile shops). Personally my automated tests have given more confidence about working system than finding defects. What have your success ratio with automated UI tests? or may be how you define success with automated UI test?

share|improve this question
2  
My opinions on automated GUI tests are here. angryweasel.com/blog/?p=7 –  Alan May 5 '11 at 13:22

7 Answers 7

up vote 25 down vote accepted

My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs.

But what they do do is two key things:

  1. They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs.

  2. They give you the opportunity to heavily re-factor or make large changes quickly and confidently.

share|improve this answer
    
Yeah what he said just shorter then me. :) –  Hannibal May 4 '11 at 11:45
    
I think it makes the break/fix cycle loads faster. –  terryp May 4 '11 at 11:49
    
+1 You shouldn't have to spend time re-exploring covered ground. You found that bug, you wrote the test for it, and we have time to spend testing unexplored avenues. –  corsiKa May 5 '11 at 22:52

The previous posters are correct - automation can never find new bugs, that can only be done by sapient testing - however automation will find regression faults, and often not the ones you are looking for. However, sapient analysis of the results of automation tests can find significant issues.

I cite an example for my last project where we had close to 1000 automated tests running through the UI every day. We had an occasion where some of the tests were failing intermittently and without any clear reason (the code around that area was stable and hadn't been touched for some time).

So when I looked at the test results, I couldn't see anything immediately. One of the test criteria was that the page title was "foo", and when I looked at where the assert had failed I could see that the actual page title contained "http 403".

Just by looking at that failed assertion, I immediately had a rough timescale for the failure, and could more quickly isolate the completely unrelated problem that was unintentionally bringing down the site for a few hours perhaps once a week.

I can't say if our test team would have found that bug or not without the regression suite, but it would have taken much longer for sure.

share|improve this answer
    
Your example is not a failure of test automation, it is an example of a poorly designed autoamted test with poorly designed oracles. I can give multiple counter examples of where randomized inputs using random test data generation and data-driven combinatorial testing found many bugs in GUI automated tests. –  Bj Rollison May 6 '11 at 15:27
    
Poorly designed test - how exactly did you work that out from the brief anecdote I provided? –  CBA May 9 '11 at 19:36
    
You're correct, that perhaps was a faulty assumption on my part. From your anecdotal statements "tests failing intermittently" and "without clear reason" I jumped to potential conclusions for these types of issues might include sync/timing/race conditions in your script, and lack of diagnosability (e.g. expect "foo", actual "http 403", and stopwatch function). My comment was not intended to disrespect your automation project, but was simply to counter your statement that automation can never find new bugs. –  Bj Rollison Jun 3 '11 at 5:41

You have to understand the following...

Automation will never find NEW bugs.

It will find bugs if your system is updated. It will find bugs if the system is down for whatever reason and you are sleeping in your bed and you have automation enabled overnight. It will send you an urgent email that you will read in the morning that the site was down for an hour or so.

It won’t find bugs that humans find through bug regression. It won’t find bugs in the system that require analysis and constant pounding. It won’t find hidden bugs. It will only find bugs that it is programmed to find.

That said... Automation is something that will help you concentrate on those hidden bugs. It will get a LOT of testing off your shoulder. Like multy browser, endless input of various combinations of data. Tidies tasks like create test data, companies, users, etc. Stress test, Load test, functionality. All that can be covered.

So success for an automated test would be the assurance that if a developer checks in code, you can be sure that a major part of the application is regressed at night when you are sound asleep. And you can be sure that it will find changes, defects which that update produced and nothing will be unnoticed unless you didn't write a test for it.

share|improve this answer
4  
I disagree with your first statement. "Regression" tests will never find new bugs, but data-driven automated tests, or automated model-based tests /will/ find new bugs. –  Alan May 5 '11 at 13:08
1  
I'll add, that in context of the original question, @hannibal's answer is correct, as the question was about GUI regression tests. I'm just wondering what to call my perf, stress, and model based automated tests that find new bugs all the time. –  Alan May 5 '11 at 13:23
    
Yeah since the question was about GUI perf tests aren't regarded in my answer. Also, in pont of NEW i meant to say that if you have a set of Tests that search ONE part of a product then the bugs that it find will always be from that part. But never from somewhere else. I don't know if that's clear enough. :) –  Hannibal May 5 '11 at 13:34
1  
If your automated tests are brain-dead scriptlets with a bunch of hard code actions/data, then I agree with your statement. However, to claim that GUI test autoamtion will "never find NEW bugs" is simply incorrect. Well-designed automated tests can and do find new bugs. –  Bj Rollison May 6 '11 at 15:25
    
Automated tests can only do, what you program them to do. Unless you develop an AI. Prove your statement and i shall give you right. –  Hannibal May 6 '11 at 18:11

One thing to consider is that in many cases your regression suites are working off of a static set of regression data or criteria. One way that regression suites can be of higher value is trying to develop heuristic oracles vs. static ones. In a talk I went to at Seaspin on 3/1 Harry Robinson talked about moving from the static test automation to a more dynamic variety. In one slide he gave an example:

Heuristic Test Oracles

  • Verify some results for the SUT

Check other results using
– simple algorithms
– consistency checks

Static oracle: sqrt(17) = 4.1231056

Heuristic oracle: sqrt(N) * sqrt(N) = N

From his presentations slides which are available on the seaspin site above. The idea behide this being if you can test a much broder series of values in this sort of manner.

share|improve this answer

I find that automated tests in general can be very useful for ensuring that older functionality isn't getting broken by new code, and I have seen new bugs in new code uncovered by old regressions tests. I try to test as much functionality as I can without the UI, and then do some additional integration tests to make sure the UI is calling the right functionality after the functionality itself has been verified. This practice results in far more maintainable tests.

I prefer to avoid UI testing whenever it is not necessary to get the information I want from that test due to the maintainability costs. In my experience, UI automation does less to reduce technical debt compared to other forms of automation because of this ongoing maintenance debt. I prefer to write more maintainable functional tests first, and then test the UI manually if I run out of time (all else being equal, that is). Of course, some projects have a higher UI test maintenance debt than others, and automation efforts should be prioritized accordingly. The two main sources of maintenance issues in the projects I worked on were UI changes and timing changes. UI tests tend to timeout if they don't see expected behavior, so one failed test doesn't block execution of the remaining tests, but there is often noise that can also lead to timeouts and false failures.

When I do finally get to the point where UI testing is the highest priority automated testing remaining, I find that some issues with UI testing can be greatly mitigated by good design. All timeout values and CSS / XPath / accessibility navigation / etc. strings should be in one place, where they can be easily configured. Although I hadn't heard the name, I've learned with trial and error to use the "Page Object" technique, and it helped a lot with our UI test design. Good communication with developers is also a must, to avoid investing a lot of time writing tests that will immediately be broken due to a UI change. Developers can also often design their code to be as testable as possible by other automated tests, so you can minimize the surface area you are targeting with UI tests to just examining the GUI logic and the interaction with the rest of the code, and not functionality testing.

share|improve this answer

Yes, our automated test suite can and does find critical bugs. Not very often but just off the top of my head I can think of two occasions when it did.

I agree with the answers above that the main benefit of automated tests is freeing up your human testers from performing repetitive tasks and so allowing them to concentrate on the really intelligent testing they're good at. I would add though that the regression tests we automated were among the most important areas of the application (most frequently used by our end users) so when the automated tests did catch bugs they were really, really important ones.

share|improve this answer

As a developer, automated regression tests have certainly saved my day several times. But a QA professional may not be aware of this. Here is the reason:

A deterministic test can find a bug only once: When I (the developer) run the test suite and see that I made a stupid mistake, I fix this bug immediately --- and no tester is going to see it. Without the test suite, the bug would make it into the repository.

From my experience, I most benefited from simple integration tests: When an apparently benefitial change in one component had an undesired effect in another component.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.