4

We are following agile methods in manual testing. We had not done any automation until sprint 10.

Our current sprint is 15.

After sprint 10, We started automation using Java, Maven, and Selenium from Sprint 1. Now we have automated suite ready sprint wise from sprint 1 to 10.

As a QA my question is, what is the best approach to run the automation test suite. I mean there is no meaning to run whole automation suite of 1 - 10 sprints on a daily or weekly basis because we change any existing functionalities rarely.

We have different features like Departments, Employees, login, register and a lot more.

So should we run the automation test suite by features when that feature is modified? OR we should schedule all test cases of automation?

We are planning to use Azure pipeline for automation with maven build but as we are not yet confirm on best approach, we did not create pipeline.

4 Answers 4

3

Regression Testing was one of the core phases of traditional waterfall development. It meant that after the (often long) development phase was finally through, all the pieces of the new system or new version were tied together, making sure everything worked. Maybe one or twice a year or once a quarter. Taking a few weeks to do it was ok for that business model. After all "the invoice has to age for 30 days anyway..." (traditional approach and example of challenge).

In Agile development the term 'regression testing' has largely lost its meaning/value. It means little more now than 'run the whole test suite for that stage'. It's not so much that regression testing has gone away - rather the opposite - it's been accepted as just part of development and testing in all 4 agile testing quadrants.

This is a piece of education and change that will often not be identified to take place on teams that come from waterfall. Stopping using a term that has long been an accepted way of how things work is really hard. Changing its meaning even harder. Both require clear communication or discussion or you'll end up with folks with a bunch of different opinions and views on what regression testing should mean today depending on their internal / external experience.

So when developers are writing Unit code and they have the tests and the code for the unit they are working on passing they then make sure all the other units tests works (this could be thought of as a 'unit test suite regression run').

When seeing if all the components work, first test any that were changed and then run all the integrated tests (same as 'regression run') in a integration testing environment. Here you might mess with the time clock and see 30 days go by in an instant...

Same for Automated UI testing, manual testing, usability testing, etc. Run any specific tests for that function and then run the whole suite at that stage.

As with programming, Agile means breaking down testing into many components.

This provides two key returns:

  • immediate feedback at the component, integration or e2e level
  • ability to fix soon, costing less, going faster ('shift left')

The need to run the whole suite should quickly drive you to automation over manual testing. When you are releasing every few weeks or days or hours you can't run a manual test suite effectively, no matter how many off-shore resources you co-ordinate.

Refer to

The Agile Testing Pyramid

and

The Agile Testing Quadrants

for a (lot) more info. You won't see the term 'regression' much there.


Final Note:

"I mean there is no meaning to run whole automation suite of 1 - 10 sprints on a daily or weekly basis because we change any existing functionalities rarely."

This another example of a process that should change. Actually yes, you do want to run the whole suite, even if you don't think there are any changes. Sometimes there are unexpected consequences to a small seemingly inconsequential change. I have seen a tiny change bring down an entire system. This is what leads to the business requiring an extensive UI test suite whether manual or automated). Following the Agile practices of comprehensive and meaningful Unit and Integration tests means avoiding the old paradigm of eventually checking everything through the UI. Given the number of devices and browsers and versions this hasn't really been practical for a while anyway. In an age of speed you definitely need a small number of smoke and UI tests.

4

It depends on the execution time of the automated test suite - if it's few minutes then it can be run after each new commit - when new changes are introduced to the code-base. If it takes longer then you can run it at the end of each sprint or even just before the new version of application in question goes to the verification / validation stage. It should compliment your specific development process and not hinder it with additional waiting time to build / release application.

I mean there is no meaning to run whole automation suite of 1 - 10 sprints on a daily or weekly basis because we change any existing functionalities rarely.

So you can say with 100% certainty that new code will not break any of existing functionality?

3
  • I am not 100% sure that nothing will break but still, it looks not reasonable to me to run the whole suite every time. Commented Sep 11, 2019 at 15:00
  • Could you elaborate on your reasoning behind this? Another solution is running partial regression suite based on some risk value matrix criteria.
    – Prome
    Commented Sep 11, 2019 at 15:06
  • 1
    As per Prome's answer it's all about how long your suite takes to run. If you run your tests in parallel (using either the inbuilt functionality of TestNG or a Selenium Grid) then large suites can become very quick runs. Tie in a machine with Jenkins to monitor the build pipeline and fire off the full regression after every build (or after any QA ready build) and it's very useful. e.g. if it takes less than 20 mins to run 500+ regression tests then it'll just complete while you to go though something manually, but it saves a lot of time as you can report back on issues immediately.
    – Klynt
    Commented Sep 11, 2019 at 15:20
4

You have automation regression suite. You are working in agile. Cost of running regression daily is close to zero (you have build agents and CI/CD working).

You can add value to developers by finding regression bugs the same day they were created. You can decrease cost of fixing bugs. Yet you say there is no sense in running it daily/weekly.

There is. And it cost you almost nothing.

3
  • +1,on cost/ value analysis.If complete suite can be scheduled to run overnight, how does it matter if it takes 2 or 4 hours? Commented Sep 13, 2019 at 0:38
  • We are running complete regression suite overnight.Many times we catch unexpected bugs in one area due to effect of a change in remotely associated module.However if we don't run it daily , it will be very hard to track it down later. Commented Sep 13, 2019 at 0:42
  • Yes you are right Michal, I agree on zero cost. Commented Sep 14, 2019 at 4:54
1

My 5 cents...

First and most important point - testing is not just the responsibility of the QA's but every member of the Scrum team. Keeping that in mind, testing in Agile should be continuous. This is applicable to Regression testing as well.

We followed the below approach to convert our manual regression pack to a continuous testing model.

  1. Identify the test scenarios required to execute to make sure that all functionalities (major) are working as expected. Lets call it regression pack for now.

  2. Now break that pack down in such a way that, we execute the tests as early in SDLC as possible. For eg: some tests can be broken into Component level tests or API tests and can be automated easily and executed right after development. Hence finding the defect very early. Devs can contribute the team in taking up the ownership of the Component and API tests automation (Mostly QAs expertise in Component and API Automation will be limited)

  3. Do a ROI(return of investment analysis) on the UI level tests before considering for automation. Again plan in such a way that some or most can be run in Dev servers as well (using mock API's or synthetically generated test data)

  4. Execute the remaining tests manually, again run the tests as early in the SDLC as possible.

So eventually your manual regression test pack (say 200 test cases) will go away and instead you will have a set of Unit/Component, API, Automated UI and Manual UI tests (say around 10 times the manual test cases) which you would be able to execute early in SDLC and find defects closer to the origin point.

So should we run the automation test suite by features when that feature is modified? OR we should schedule all test cases of automation? -- very good question. In our team we follow a modular development approach, where the features are isolated if possible in such a way that it can be developed, tested and even pushed to live independently. This can help improve your time to go live.

1
  • Thank you for your inputs, moduler development approach gave me a very good hint. Commented Sep 14, 2019 at 4:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.