1

I have 120 test cases in my project. All those test cases are running on a live site. Here I configured all these 120 test cases in a single suite (TestNG.xml) file. It's taking 4 to 5 hours to complete all these test cases but sometimes some test cases are going to fail but when I run them individually they pass. So here I am doing anything wrong by clubbing all 120 test cases in a single suite in a TestNG.xml file.

Can anyone guide me the best way to deal with executing large number of test cases?

Environment : Selenium Webdriver,Maven,TestNG and Jenkins

Here It is the sample TestNG.xml file

<suite name="SUITE">    
<test name="Test 1"> 
    <parameter name="folderName" value="Test1"></parameter> 
    <classes> 
        <class name="com.org.Test1" />
    </classes>
</test>
 .
 .
 .
 . 
 <test name="Test 120"> 
    <parameter name="folderName" value="Test120"></parameter> 
    <classes> 
        <class name="com.org.Test120" />
    </classes>
</test>
</suite>
1

With the caveat that this answer isn't Selenium specific.

You aren't necessarily doing anything explicitly wrong by having a 'large' number of tests running in one suite. For example for back end testing some of the suites I run have millions of test cases. One notable test suite I support has over 200 million tests as part of it's run(Granted this is back end testing, data driven and not in selenium).

But, a very important but, we don't run all 200 million in parallel because testing has to exist, for the most part, in the real world and very few systems can even run that many tests at once let alone support that many actions being tested against them at once. I don't see you running these in parallel so I'm assuming that you are indeed running these in serial. Even so, a large number of serial actions against a product, system or site can impact performance in a way that can cause test failures. Additionally your tests may be interacting with each other in unexpected ways (changing the expected state behind the scenes.)

There are a couple of things you can do here to narrow down your issue:

  1. Do the same tests always fail when run in the larger test suite? If yes then you potentially have a couple of culprits:
    • Try changing the order of your tests. Try to narrow down if your tests are not truly atomic and automated. This should help you identify the case where a specific test or tests are impacting other tests unexpectedly. It's worth pointing out, if your tests are doing what they should this could be a bug in the system itself.
    • Try spacing out your tests by adding waits. It's possible that a specific test is putting your system under load and causing issues. Spacing out actions can assist in identifying whether this is a specific test problem or a larger load problem. This leads nicely into...
  2. The thing you are testing may not be capable of handling the load you are sending it. If you are testing in a staging environment the load expectations may be much lower than a production environment. If you are testing your production environment you may have specific actions that decrease your system's performance. You may just be hitting it too hard or too fast.
    • Typically speaking, Selenium is not a tool for testing load or performance and if load/performance are getting in the way of functional testing you should rate limit your tests in some way to see if this improves the results of your tests.
    • Your tests may be doing actions behind the scenes that cause issues. If you authenticate, for example, every time you do an action you could be unintentionally causing very high load for your tests. Actions that make sense to be memoized(that aren't under test, that are expensive, whose data doesn't change over the course of your test, and that you use often) should have their data chached in some way.

Finally - this may be an underlying bug that occurs when disparate actions are performed against your product. The best way to catch something like this would be to identify the components of your test suite that trigger this (if test X always fails, figure out how many or what other tests can repeatedly cause this.) Bugs like this are very difficult to find and even harder to prove, you may (while keeping this issue in mind for further triage and research) have to do any of the steps above to test the individual functionalities of your product.

1

I am expecting it as not an issue, in my experience most of the times few tests fails on bulk execution and they will pass on individual execution. May be its sort of sync issue, i hope.

If you think its due to large suite then you can split it into multiple suites and call those in different builds as you are using Jenkins. If you are using multiple VMs/Machines then each build on different VM or slave with different suites, helps to complete execution in much less time.

0

If tests are failing when run in a batch, most likely they are not independent tests, but depend on the state of the System Under Test (SUT) to pass.

For each test, make sure you are in control that the SUT actually is in the expected state (users with rights, data present, etc) as part of your test in the setup phase of your test. Do not make any assumptions on what's already there. Equally important, reset whatever changes your test has made to the system (in the teardown, or cleanup, phase of the test).

Finally, 4-5 hours is a really long time for an automated test suite to run. Are you sure you are automating the right things at the right level?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.