Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Once an automation test is created, how do I test it and make sure the test does what its supposed to do? I can easily test the scenarios where test passes, but how to go about the ones that fails due to issues in site? Issues such as - an element missing, click not working etc. All I could think was to mock the HTML to simulate the condition to test and run the test in that local html.

Does any one else have better idea?

share|improve this question
1  
To clarify: are you talking about making a test fail gracefully (as in the test finds a problem and needs to fail)? Or the testing a test user246 described? –  Steve Miskiewicz Jun 23 '12 at 0:17
    
I was looking at the graceful failure. –  A.J Jun 26 '12 at 18:21
add comment

4 Answers 4

up vote 3 down vote accepted

I think you are asking whether you need meta-tests (automated or otherwise). Of course, how do you know that a meta-test is correct? Here is what most developers do: if something feels risky, they test it; otherwise they trust that their code is correct. If something that they trust turns out to have a bug, they try to re-calibrate their judgement. You should do the same thing.

Personally, I would not go through the trouble to mock the HTML. If you have a site you can modify, you might try introducing bugs to see if your tests find them. Of course, you want to introduce realistic bugs, and it may not be easy to decide which bugs are realistic.

Edit

We test in order to reduce risk, but testing to excess can be ineffectual and even destructive. If you choose to write automated tests, you are subject to the same decisions that developers are subject to. Most developers do not choose to test every line of code. This is a good thing. Every line of automated test requires additional development effort and additional ongoing maintenance and will not necessarily find more bugs.

There is no hard and fast rule for deciding when to test and when to not test. The decision rests on your own experience, your confidence as a developer, your familiarity with the subject matter, your confidence in the code that you depend on, the maturity of your product, and other factors. Rather than looking for a hard-and-fast rule to follow dogmatically, try something that feels right, pay honest attention to your results, and re-calibrate accordingly.

Here is a rule of thumb that you could try. If testing something requires a huge effort, question your assumptions. You may be trying to test it the wrong way, or the thing you are trying to test may be more complicated than it needs to be, or you might even decide that the risk of it being wrong does not justify the cost of testing it.

share|improve this answer
    
I don't know what meta-tests are. Can you explain what that is? And I can't modify the site to introduce the bug. My question more about how will I make sure all parts of my automation script (which is in JAVA) gets tested. Does that make the question clear? –  A.J Jun 22 '12 at 20:40
    
I was making an absurdum ad infinitum argument. A meta test is a test of a test, which I think was the subject of your question. If you test your test by mocking HTML, how do you know the mocking code is correct? If it has bugs, that could invalidate your meta-test. So will you write a test for your mocking code too? And then how do you know whether your test of your mocking code is correct, unless you test it too? A some point you need to be able to write correct code without writing tests; otherwise, you should not write code. –  user246 Jun 22 '12 at 22:29
1  
@user246 "A some point you need to be able to write correct code without writing tests; otherwise, you should not write code." Taken to extremes, this would suggest that software quality assurance is unnecessary. However, since I'd like to think I'm not a parasite, I must ask: How do you determine where to draw the line? How many layers of tests and meta tests are appropriate? The answer seems to be at the heart of the OP's question. –  user867 Jun 25 '12 at 6:42
    
@user867 I will update my answer to address your concerns. –  user246 Jun 25 '12 at 14:50
    
@user246 Excellent! You have my upvote. –  user867 Jun 26 '12 at 4:43
show 1 more comment

You can also take the mutant build approach: insert bugs in when generating the build and see if your coverage find any of them. The higher the number the better your coverage is, at least theoretically speaking :).

share|improve this answer
add comment
  • "I can easily test the scenarios where test passes", my experience shows the opposite. Take for example a test where the return value was initialize to "Pass" at the beginning (really bad practice) and a bug in the test code send it to the end of the test before something was actually tested.
  • We usually add a set of unit-tests for the core services such as utility functions.
share|improve this answer
add comment

When I'm automating a test, I often make the test wrong in some small but meaningful way, then run it with the expectation that it will fail. For example:

  • If the system is supposed display "42" in the "meaning-of-life" field, I'll change the the test to assert that it displays "43".
  • If the system is supposed to display "42" only when the current user is "Ford Prefect," I'll change the test to log in as "Zaphod Beeblebrox".
  • If the system is supposed to display "42" only after Ford visits the "Restaurant at the End of the Universe" page, I'll change the test to omit the "visit the restaurant..." step.

I'll do one of these at a time. I'll make the test wrong, run it, and notice whether it still passes. If the test now fails, and it makes sense to interpret the failure as due to my change, I'll conclude that the original test got that detail right, and change it back.

If the test still passes, that means the detail I changed does not have the effect I wanted, so I'll need to do some debugging.

If the test fails, but the failure isn't clearly attributable to the way I broke the test, I'll need to do some debugging.

Once I've tested the test in those ways, and once the system passes the test, I treat the test as being reasonably reliable until I discover some reason to doubt it (e.g. the test reports a failure, which might indicate a problem either with the test or with the system; or the test reports success when I have reason to believe the system or its environment is broken).

share|improve this answer
    
+1 for testing your code and making unnecessary Hitchhiker references. Always keep a towel at your desk. –  Steve Miskiewicz Jun 25 '12 at 23:04
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.