Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm working on automated UI tests in Microsoft's Coded UI.

However, I noticed that my tests are only succeeding occasionally.

For example, I ran the same test 4 times. It succeeded once, and failed the other 3 times.

I know that my question's a bit open-ended, but what does the haphazardness of my tests indicate?

share|improve this question

closed as too broad by corsiKa Jul 2 '13 at 19:42

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

welcome to SQA. In my experience, when you ask someone to help you with a programming problem, it helps to explain what your program does and what kind of errors you are experiencing. Perhaps you could try adding those details to your question. – user246 Jul 2 '13 at 13:42
Can you fill us in on what the process is that you're testing? Also, is this a web application or desktop app that you're writing tests for? – Daniel Jul 2 '13 at 14:35
Hi, Kevin, could you give us more information on the errors you're getting? Right now there isn't enough information to make any meaningful suggestions beyond what BJ has said. – Kate Paulk Jul 2 '13 at 16:26
Hi Folks - sorry I didn't provide more of a description of my problem. I'm working on a C# web app that consists of filling out and querying against form data (Name, Date, Address, etc.). I found Bj's answer useful, so I selected his solution. In the future, I'll be more detailed regarding my problem. Thanks for the welcome. – Kevin Meredith Jul 2 '13 at 19:01
Hey Kevin. Thanks for being so open to feedback on your question here. Even though the question is answered, I'm going to place it on hold so that future visitors know that in its current state, it isn't exactly a great fit for this format. Thanks for being such a sport about everything! – corsiKa Jul 2 '13 at 19:41
up vote 9 down vote accepted

The symptoms sound indicative of what we would classify as an unreliable test; meaning the test does not produce the same result when executed against the same exact product code. UI automated tests are notorious for being unreliable for a variety of reasons.

Assuming no change in the code being tested and no variability in the test (especially with UI automation) I would first suspect synchronization / race conditions between your test code and the product behavior at runtime (e.g. your code is expecting the machine to be in a particular state and the app response is slower than expected). It is also possible that your test is waiting on a particular state, or response from the system/service and the test times out before the system is ready/responds. These are perhaps the most common issues with unreliable automated tests, but certainly not the only ones.

But, an unreliable test could also point to an issue in the product (performance, inconsistent behavior, etc).

In my team when we come across unreliable tests we open a bug against the test and debug the test to identify the cause of the issue and address it appropriately. On a few rare occasions we changed the bug from test code defect to product code defect.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.