When and how to do regression testing?

Question

I am learning selenium. I am trying to get into more and more automation.

Scenario (example):

We are working in agile 2-week sprints. We have a release after every 3 sprints.

In our framework, we have used the hybrid approach (a combination of data-driven and POM). Every test class has 40 test cases and we have 20 test classes. In a total of 3 sprints, there are 800 test cases out of which we need to select 500 test cases for regression.

Question: which method we should use

1. Using TestNG annotation groupBy to give a name as regression and then create a testng.XML file (or regression.XML) and pass with the group. If yes then how to declare in our XML file?

2. Is there any better approach?

Answer 1

The best method in my opinion will be to explore Groups in TestNG. You can mark the 500 Test Cases in Regressions as below :

@Test(priority=1, groups={"Regression"})
public void testCaseOne() {

}

In your testNG XML, include Group Tag to Run these testCases.

First mark all the Test Cases as per the need and then generate testNG XML by Right-Clicking on Project -> TestNG -> Convert to TestNG. The resultant XML should automatically have Groups tag to give you control over Test Execution.

Answer 2

Regression Testing has always been a key concept for manual testing. Due to the nature of needing people to do it, with all that involves, it is important to identify a subset of key features to make sure they still work. This involves a lot of time and effort to create and maintain. All of which is time not writing application or automation code.

Companies that have moved to more automated testing have found the term 'regression testing' to be largely irrelevant. On of the key features of automation is that is runs fast and can be scaled. A consequence of this is that you should just run all your tests when making a change.

This doesn't necessarily mean running all tests as you make each small change. While making a change you will want to run relevant test layer suites. So for Unit testing you want to run all the Unit tests. They should be running in a few seconds to give the immediate feedback you need. This is achievable when you have stubbed and mocked ALL external services, database, filesystem, etc. Before merging the change to master however, the developer should run the full suite of UI tests.

In more traditional environments this will seem waaaaaaaaay too big a burden, this is because they have not adopted agile testing and still have an ice-cream cone for their test pyramid which emphasis on extensive and exhaustive UI testing. They still have factors such as '18 hour test suites' (or longer) that are essentially a liability for modern agile development. Continuous integration and getting used to branches running on your CI server is also a key part of this change. Ultimately part of the fix is also to monitor production releases and to perform Canary, AB and Feature flag releases.

Always question the value of a process and remember

  Individuals and interactions over processes and tools

  Working software over comprehensive documentation

and

Automation code, by itself, does not improve application code quality

Answer 3

Here is the Sample XML file code

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Sample Suite">
  <test name="testing">
    <groups>
      <run>
        <include name="Regression"/>
      </run>
    </groups>
    <classes>
       <class name="com.example.group.groupExamples" />
    </classes>
  </test>
</suite>

For more details checkout TestNG groups

Answer 4

You can declare the groups as

<groups>
<run>
<include name="group1"></include>
<include name="group2"></include>
</run>
</groups>

Then in the class file declare as

<classes>
<class name="packagename.ClassName"/> 
<class name="packagename.ClassName" />
</classes>

If you are passing any parameters then declare as

<parameter name="browser" value="chrome"></parameter>
<parameter name="url" value="https://localhost"></parameter>

Answer 5

you can go with BDD-cucumber .cucumber will provide you the custom tags by which you can easily divide your regression suite to multiple groups such as, @regression @sanity @high-level

Example suppose i am having 1000 test cases in my regression and i have to do the only high-level testing or sanity check so while writing the feature file you can give the custom tag to each scenario and your runner class will be:

@RunWith(Cucumber.class) @Cucumber.Options(format = {"pretty", "html:target/cucumber"}, tags = {"~@regression"}) public class runTest { }

Feature file:

https://i.stack.imgur.com/wPso0.png