Wanted to know common practice for determining which tests to be executed & which to be skipped in TestNG/RC?

Question

We are using Testng with RC. Would want to know a common/practical/generally used way to determine which tests to be run & not run.

Take to scenario - In one test suite I have 3 modules - A, B & C. In each of the modules, there are 5 - 6 tests. The tests have been created & run fine. But as my tests would increase I may want to skip a few of tests in either of the module. I want to only run A3, A5, B1, B2, C3,C4 & C5. How would I implement these settings? Ways that I could think of -

1.Should these settings(tests to be run) be done from excel file. Say the sheet has a list of all the test case(title only) & only ones that are flagged "yes" should be executed. 2.Should these settings be configured from the testng.xml itself? If yes then how? I am aware that I can create groups & do that. But is that how is done? The overall idea is not to touch the tests or play with the annotations to miss the tests. If 1 is the answer then how would the excel file communicate with the Testng to tell it what test to be run & which one are not to be run? Or if there is another way(surely there would be) to acheive this.

Please provide some inputs on the issue, let me know if something is not clear. Have posted the same question on stackoverflow as well.

Regards Tahir

Answer 1

Typically you are going to want all of your tests to run upon checkin to ensure that no changes have been made which will cause a regression or integration error. Typically it is considered a best practice to run these on a build machine rather than the developers machine, so that they are able to get back to productive work. The developer should know which tests to run based upon the changes that they made.

When the build is run, unit tests should run, then integration tests, and finally acceptance tests. If any of these tests fail, then the developer(s) who made the last checkin(s) should immediately begin on working to fix the build. I added the plural case to those developer and checkin because there might be many fixes checked in since the last automated tests ran to completion.

If you really had to do what you proposed, I would suggest creating a custom runner in your testing framework, which would have the logic in it to determine which tests to run; however I see that quickly getting cumbersome for the developers to change, and most importantly keep up to date -- I see lots of merge conflicts with the configuration file as well, since developers will actively be working on many different aspects of the system.

In short, I'd follow the best practice of having the developers run the tests they think most pertain to the areas they are working on, then running all tests on an external system upon checkin.

Answer 2

Welcome to SQA, Tahir.

My answer comes from some past experience with JUnit; although it is similar, I do not have direct experience with TestNG.

I agree it is preferable to specify which tests you want to run without having to change any source code. You mentioned starting with an Excel file containing the test titles. I do not know what you mean by "title", but I would start with a CSV file containing the names of the tests, either at the class level or the method level, depending on how granular you need to be.

A disadvantage of this approach is that adding a test may require changing things in two places, which opens an opportunity for maintenance errors, as Scott mentioned in his answer. Using a CSV file -- rather than an Excel file or an XML file -- may help with the maintenance burden. Depending on how your code is organized, it may also be possible to write a program that scans all your test source code and automatically adds names of new tests to the file.

If you have a lot of tests, then as you recognize your usage patterns, you will want a grouping mechanism in your CSV file. You would need to decide on a policy for how newly-added tests should be grouped.

I see TestNG provides a way to list the tests you want to run in an XML file. If you do not want to write a custom test runner, you may be able to write something to convert your CSV file into the corresponding XML file.

Finally, I think there are two distinct use cases: defining multiple (and possibly overlapping) suites of tests that should run on a regular bases, and specifying an arbitrary, ad-hoc set of tests. What I described above is suited for the first use cases. For the second use case, I may also want a way to specify the tests you want to run from the command line. I wrote a custom test runner for JUnit that let me specify test classes (or individual test methods) from the command line. I used that option frequently.

Answer 3

Have a look at the groups documentation:

http://testng.org/doc/documentation-main.html#methods

Specifically exclusion groups

5.3 - Exclusion groups

TestNG allows you to include groups as well as exclude them. For example, it is quite usual to have tests that temporarily break because of a recent change, and you don't have time to fix the breakage yet. 4 However, you do want to have clean runs of your functional tests, so you need to deactivate these tests but keep in mind they will need to be reactivated.

A simple way to solve this problem is to create a group called "broken" and make these test methods belong to it. For example, in the above example, I know that testMethod2() is now broken so I want to disable it: @Test(groups = {"checkintest", "broken"} ) public void testMethod2() { } All I need to do now is to exclude this group from the run:

<test name="Simple example">   
   <groups>
     <run>
       <include name="checkintest"/>
       <exclude name="broken"/>
     </run>   
   </groups>
   <classes>
     <class name="example1.Test1"/>   
   </classes> 
</test>

This way, I will get a clean test run while keeping track of what tests are broken and need to be fixed later.

Note:  you can also disable tests on an individual basis by using the "enabled" property available on both @Test and @Before/After

annotations.

I'de suggest reading the rest as well.