Fitting regression testing in a Agile/ Scrum development cycle

Question

What is the best way to fit regression tests into a Agile/ Scrum development cycle?

I've been working as the sole QA person in a 4 person Scrum team developing a new web client in jQuery. I am finding a lot of my time is taken creating and running manual test scripts on the new stories and bug fixes during each sprint.

In some part regression will be covered by unit tests from the developer side and automated tests from the QA side.

Where regression testing fits in the cycle or is it enough to run unit and automated tests?

Answer 1

Ideally there should be a regression or "hardening" sprint before the project ends, and in the dream world, a hardening sprint every few cycles.

I'd suggest that automated tests of the steel thread of the sprint be developed as the sprint progresses, and any bug fixes get an automated test as well. This way, you build your automated regression as the development progresses with much less pain than manually re-running each manual test script for each new sprint. This also helps to deal with the common real-life situation where the regression/hardening sprints don't happen.

By focusing automation on the combination of steel-thread and known breakages, you're likely to be effectively regressing the most used and most fragile areas all through development, and can use any "found" time to add automation for any other areas you consider potentially risky.

I've also found it helps to prioritize potential breakages according to likely use frequency and impact - something that's likely to happen commonly and would take the application down if it broke should have a regression test to ensure that any breakage is caught before deployment.

In short, regression should be happening throughout the cycle as part of automated and unit tests, and continuously expanded to cover any new issues that arise as well as new steel-thread stories.

Answer 2

I've worked as a solo tester on a small agile team as well, and I've found the two things that help most with regression testing are automation and risk-based test prioritization.

Having an automated test suite with good coverage is definitely very helpful. However, I wouldn't recommend relying entirely on automated tests for regression, as there are some types of bugs that automated tests aren't particularly good at detecting.

A time-efficient way of complementing the automation effort with manual testing is to pick out a subset of manual tests based on a risk analysis of the system. Have a think about the areas of the system that are most likely to be impacted by the changes for the sprint, and only target those areas with your manual testing.

For best results, I usually use the manual test cases for those at-risk areas as a guide for my exploratory testing.

I have found this approach results in a much shorter regression testing time towards the end of each sprint, and it has been enough for the level of quality required for our projects. However, it does leave a little room for minor to trivial bugs to slip by in low-risk areas because the low-risk areas are entirely dependent on the automated tests to detect bugs. This may not be acceptable for some project standards, so just be aware of that risk.

Answer 3

I cheat, and break a couple of rules of "Agile" development.

Firstly I make an arbitrary automaton target of 80% of all tests part of the process that must be followed, at a stakeholder level. This number is based on experience, and forces a balance between manual testing and writing automation. It also forces appropriate resource allocation from a budgetary prospective.

Why 80%?

I don't know where that number originally came from, but as it turned out, on the first project where I was trying to hit that magic 80%, we had to work pretty hard to achieve that goal. The payback was, that as the project developed, we didn't need to manually regression test as much, and we could run 80% of our entire test suite every day. On my last project we hit 100% across the board. So yes, 80% is just a made up goal, but it is one that seemed to work for us. :-)

When did we actually do the automation work? Well normally we used the first few days of the iteration to catch up and automate the tests for the features that were just delivered at the end of the last iteration.

So I cheat by

1. Changing the rules of the game (the process) and

2. Building tools to help accelerate the work as quickly as possible.

Answer 4

According to my view, "Scrum is the best agile approach. In scrum, each sprint produces an increment, which is a potentially releasable product. Here, each increment must satisfy all acceptance criteria and pass the different categories of tests. Regression tests are a tiresome activity, especially in an agile process, which is characterized by nonstop changes and frequent deliveries. If project ‘Cost’ affect the project then we can perform Regression manually. If not then automation is best to handle regression. Selenium, Microsoft Test Manager etc. are examples of tools that the team can use to automate regression tests. "

Answer 5

Our team deals with this by making a task for manual testing for every user story, so it gets budgeted and estimated along with all the other tasks. However, the goal is to automate as much as possible to reduce the load for manual testing (possibly to zero, e.g., you might be able to automate everything you can think of with an API). When making a decision to automate or run tests manually, the technical debt incurred by failing to automate up-front should always be a consideration. I personally like to always budget for exploratory testing on most user stories even if I think I can automate 100% of the tests, just in case I come up with some creative ideas later in the sprint.

Answer 6

Ideally, you should be including the test automation (both unit and acceptance tests (with the latter becoming part of your regression suite) as a part of what needs to be in place for an item to be DONE. It's all part of being ready to ship at the end of any sprint.

That might mean you'll have to get some help from your teammates to complete that automation before items can be accepted as done, since in my experience a 4:1 ratio places a lot of load on the 'tester'. If you wait to try and do them later 'when there is time' you'll likely fall further and further behind because often there just isn't much time for that.

Then get your CI system setup so that the regression tests run on a regular basis (I'd recommend nightly)

BTW: a great definition of DONE I heard the other day was "never have to think about that story again"

See this great talk by Ken Schwaber (in three parts) that covers a lot of this and makes some very important points.

Answer 7

With long term projects (2 to 3 years already completed) and if there are 100's of old test cases, we are doing the following for 'Master' Regression and Iteration Regression

1. Master Regression - would be a big set of test folders with merged test cases under each functonality and the new functionality or changed functionality will update existing test cases

2. ITERATION Regression - After every iteration ends, we are doing a manual regression of the stories/test cases in the next iteration. We test all the functionality of the previous iterations. In the upcoming iterations we do the iteration + 1 testing.

I understand that this involves more effort every iteration, however, with projects that have releases every 2 months or so, and if there are only 3 iterations for dev/testing, then I find this kind of approach is recommended.

Additionally, we use the 4th iteration as a regression one.

Answer 8

Kate has some good advice there, though its hard to get that hardening sprint you can bring it up during the retrospectives. What I'd suggest as well is try to schedule some time for a little regression testing each sprint to keep up on the work. Try a few acceptance tests from previous sprints and see what you can do, that way you will have a better feel all along on how things are progressing.

Answer 9

Automation is a good option in scrum or agile development only after some baselined workable product is rolled out typically after one of 2 sprints are over ,because till then requirements are quite fragile and attempting automation is not a good option, till then manual testing /acceptance tests at unit level, integration level based on either behavior driven or use case driven/model driven methodologies supporting TDD,do provide appropriate test coverage and defect identification. However once we have a workable product,we have some maturity in the test cases, they thought of as ideal and effective candidates for automation against which we can define priorities as defined by the product owner and start working further into building regression suites. so as the functionalities build up with subsequent sprints we get a more robust automation regression cycle coming up in shape and over all effectiveness and efficiency of the testing effort increases exponentially.

Answer 10

This is really a good question that I've been thinking of. I'd like to add something here.

A simple answer to the question, and to the point that scrum means everything should be done is done in the end of the sprint, is automation should be done together along with other tasks in the sprint.

But that is too realistic. In the real world, id all depends. There would be a couple of factors come to play, such as test/automation resource availability and automation ROI.

If you have dedicated testers in the scrum team, or if not everyone can write test automation, that means automation sometimes became a critical path in the sprint. If team depends on the automation being written to do regression for the current sprint, it would require good flexibility on allocating dev resources to automation development. This is something not every scrum team could do in real world. It may end up that some automation work is put into backlog and became tech-debt, which has to be handled in later sprints.

On the other hand, if a new product or new features in product is not in a stable state. Significant changes would be introduced multiple times, it is not wise to automation such areas immediately. Manual test or exploratory test would be more efficient to get the product shipped. And automate later could save a lot of automation maintenance effort. If team knows for sure this is going happen, automation early may not be a good idea. This is the nature of automation development.