I joined a new company that is working on network equipments like Routers/switches.

We have a product(a network switch) in end stage of development. This supports MPLS, OSPF, IS-IS and BGP protocols. We have an existing regression testcase suite to run on daily builds.

My question is related to analyzing the test results. We have seen that some of the testcases have different results (Pass/Fail) when run on the same build and with same inputs.

What do you make out of it?

  1. The product is very buggy.
  2. The results do vary for testcases that involve network products. So don't bother??

Ah, the bane of every tester.

I should first remind you that you should control every factor to make them repeatable - which is a lot of effort. There's the environment (OS's, machines, connections/interfaces between them, then the application under test). Test fixtures are vitally important - setting the whole environment to a fixture in this case.

But you may find that instead of controlling all those factors, if you can - run the same set of tests a number of times and take a probabilistic view to comment on the results.


I'd say neither; you need to understand why you're getting different results for some of the tests, and either improve the tests, so they always pass/fail as they're supposed to, or remove them from your regression bucket.

Improving the tests might mean changing the way you do regression testing; one common reason for things to fail sometimes, but not others, is because there's some sort of interference from other testcase(s) running at the same time, but if you run the test by itself, it runs successfully. Another possibility is that order matters, and the test case needs to reset some system option or something similar that some other testcase has changed during its run.

Having testcases in a regression bucket that you ignore isn't very useful: regression testcases are there to tell you whether you broke one thing while fixing/changing something else, and if you have a regression test case that works 90% of the time, it failing doesn't tell you anything useful, unless you want to dig into the output every time to figure out why it failed.

Now, depending on the type of testing you're doing, this approach may differ; if you're doing load/stress testing, for example, a probabilistic approach to analyzing test case failures may be appropriate, as you'd expect that at least some portion of the time, the system would be too busy to, say, respond to a new packet coming in.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.