I'm working on automated UI tests in Microsoft's Coded UI.

However, I noticed that my tests are only succeeding occasionally.

For example, I ran the same test 4 times. It succeeded once, and failed the other 3 times.

I know that my question's a bit open-ended, but what does the haphazardness of my tests indicate?

  • 1
    welcome to SQA. In my experience, when you ask someone to help you with a programming problem, it helps to explain what your program does and what kind of errors you are experiencing. Perhaps you could try adding those details to your question.
    – user246
    Jul 2 '13 at 13:42
  • Can you fill us in on what the process is that you're testing? Also, is this a web application or desktop app that you're writing tests for?
    – Daniel
    Jul 2 '13 at 14:35
  • 1
    Hi, Kevin, could you give us more information on the errors you're getting? Right now there isn't enough information to make any meaningful suggestions beyond what BJ has said.
    – Kate Paulk
    Jul 2 '13 at 16:26
  • Hi Folks - sorry I didn't provide more of a description of my problem. I'm working on a C# web app that consists of filling out and querying against form data (Name, Date, Address, etc.). I found Bj's answer useful, so I selected his solution. In the future, I'll be more detailed regarding my problem. Thanks for the welcome. Jul 2 '13 at 19:01
  • 1
    Hey Kevin. Thanks for being so open to feedback on your question here. Even though the question is answered, I'm going to place it on hold so that future visitors know that in its current state, it isn't exactly a great fit for this format. Thanks for being such a sport about everything!
    – corsiKa
    Jul 2 '13 at 19:41

The symptoms sound indicative of what we would classify as an unreliable test; meaning the test does not produce the same result when executed against the same exact product code. UI automated tests are notorious for being unreliable for a variety of reasons.

Assuming no change in the code being tested and no variability in the test (especially with UI automation) I would first suspect synchronization / race conditions between your test code and the product behavior at runtime (e.g. your code is expecting the machine to be in a particular state and the app response is slower than expected). It is also possible that your test is waiting on a particular state, or response from the system/service and the test times out before the system is ready/responds. These are perhaps the most common issues with unreliable automated tests, but certainly not the only ones.

But, an unreliable test could also point to an issue in the product (performance, inconsistent behavior, etc).

In my team when we come across unreliable tests we open a bug against the test and debug the test to identify the cause of the issue and address it appropriately. On a few rare occasions we changed the bug from test code defect to product code defect.

Not the answer you're looking for? Browse other questions tagged or ask your own question.