Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

What is the best way to fit regression tests into a Agile/ Scrum development cycle?

I've been working as the sole QA person in a 4 person Scrum team developing a new web client in jQuery. I am finding a lot of my time is taken creating and running manual test scripts on the new stories and bug fixes during each sprint.

In some part regression will be covered by unit tests from the developer side and automated tests from the QA side.

Where regression testing fits in the cycle or is it enough to run unit and automated tests?

share|improve this question
    
Great question. I'll be watching the answers to this carefully to pick up tips :) –  testerab May 18 '11 at 7:44
    
I am in a small AGILE development team with no automated testing. We use Scrumwise to keep up with development-the above answers address the question with automated tests and hardening testing at the end of the project. I got dinged in regression testing after one sprint. Should I do regression testing at the end of each sprint? Thanks, /.jp –  user3033 Sep 2 '12 at 17:11

9 Answers 9

up vote 7 down vote accepted

Ideally there should be a regression or "hardening" sprint before the project ends, and in the dream world, a hardening sprint every few cycles.

I'd suggest that automated tests of the steel thread of the sprint be developed as the sprint progresses, and any bug fixes get an automated test as well. This way, you build your automated regression as the development progresses with much less pain than manually re-running each manual test script for each new sprint. This also helps to deal with the common real-life situation where the regression/hardening sprints don't happen.

By focusing automation on the combination of steel-thread and known breakages, you're likely to be effectively regressing the most used and most fragile areas all through development, and can use any "found" time to add automation for any other areas you consider potentially risky.

I've also found it helps to prioritize potential breakages according to likely use frequency and impact - something that's likely to happen commonly and would take the application down if it broke should have a regression test to ensure that any breakage is caught before deployment.

In short, regression should be happening throughout the cycle as part of automated and unit tests, and continuously expanded to cover any new issues that arise as well as new steel-thread stories.

share|improve this answer
    
This is some good advice, you should automate as much as possible. If you get to the end of a sprint and you have to manually testing new features you should add a post-it note to a board somewhere so that you can see when you are building up a manual regression suite, this will help indicate when you need a hardening sprint to reduce the manual testing debt you've acquired of the the previous sprints. Also, have the whole team do some exploratory testing before the end of the sprint, this will catch the fringe cases and usability issues. –  stuartf May 17 '11 at 22:06
3  
If you have to do extra sprints before you can ship, I think there's something wrong with your definition of DONE. In scrum you should be able to ship (if it makes sense for the business) at the end of very sprint. IMHO this is the wrong answer because you are starting to fallback on the older waterfall habits of 'testing in the quality at the end' instead of making sure that as each item is done you are truely finished with it and don't have to think about it again. –  Chuck van der Linden May 20 '11 at 19:10
    
Automation should itself be a task in Kanban/Scrum. It's possible to get to the end of a sprint and not have all tasks on the sprint done but supposedly, while you're running the sprint, you prioritize things so that stuff for any one user story is not half done. That allows you to ship SOME. Anything left over is backlogged to the next sprint, that includes automation. The problem I've run into is that automation, while backlogged, is then never prioritized as a "top" item ever again and, therefore, never gets done. This is a project management problem in that case. –  TristaanOgre May 20 '11 at 20:12

Kate has some good advice there, though its hard to get that hardening sprint you can bring it up during the retrospectives. What I'd suggest as well is try to schedule some time for a little regression testing each sprint to keep up on the work. Try a few acceptance tests from previous sprints and see what you can do, that way you will have a better feel all along on how things are progressing.

share|improve this answer

Our team deals with this by making a task for manual testing for every user story, so it gets budgeted and estimated along with all the other tasks. However, the goal is to automate as much as possible to reduce the load for manual testing (possibly to zero, e.g., you might be able to automate everything you can think of with an API). When making a decision to automate or run tests manually, the technical debt incurred by failing to automate up-front should always be a consideration. I personally like to always budget for exploratory testing on most user stories even if I think I can automate 100% of the tests, just in case I come up with some creative ideas later in the sprint.

share|improve this answer

Ideally you should be including the test automation (both unit and acceptance tests (with the latter becoming part of your regression suite) as a part of what needs to be in place for an item to be DONE. It's all part of being ready to ship at the end of any sprint.

That might mean you'll have to get some help from your teammates to complete that automation before items can be accepted as done, since in my experience a 4:1 ratio places a lot of load on the 'tester'. If you wait to try and do them later 'when there is time' you'll likely fall further and further behind because often there just isn't much time for that.

Then get your CI system setup so that the regression tests run on a regular basis (I'd recommend nightly)

BTW: a great definition of DONE I heard the other day was "never have to think about that story again"

See this great talk by Ken Schwaber (in three parts) that covers a lot of this and makes some very important points.

share|improve this answer

I've worked as a solo tester on a small agile team as well, and I've found the two things that help most with regression testing are automation and risk-based test prioritisation.

Having an automated test suite with good coverage is definitely very helpful. However, I wouldn't recommend relying entirely on automated test for regression, as there are some types of bugs that automated tests aren't particularly good at detecting.

A time-efficient way of complementing the automation effort with manual testing is to pick out a subset of manual tests based on a risk analysis of the system. Have a think about the areas of the system that are most likely to be impacted by the changes for the sprint, and only target those areas with your manual testing.

For best results, I usually use the manual test cases for those at-risk areas as a guide for my exploratory testing.

I have found this approach results in a much shorter regression testing time towards the end of each sprint, and it has been enough for the level of quality required for our projects. However, it does leave a little room for minor to trivial bugs to slip by in low-risk areas because the low-risk areas are entirely dependent on the automated tests to detect bugs. This may not be acceptable for some project standards, so just be aware of that risk.

share|improve this answer

I cheat, and break a couple of rules of "Agile" development.

Firstly I make an arbitrary automaton target of 80% of all tests part of the process that must be followed, at a stakeholder level. This number is based on experience, and forces a balance between manual testing and writing automation. It also forces appropriate resource allocation from a budgetary prospective.

Why 80%?

I don't know where that number originally came from, but as it turned out, on the first project where I was trying to hit that magic 80%, we had to work pretty hard to achieve that goal. The payback was, that as the project developed, we didn't need to manually regression test as much, and we could run 80% of our entire test suite every day. On my last project we hit 100% across the board. So yes, 80% is just a made up goal, but it is one that seemed to work for us. :-)

When did we actually do the automation work? Well normally we used the first few days of the iteration to catch up and automate the tests for the features that were just delivered at the end of the last iteration.

So I cheat by

  1. Changing the rules of the game (the process) and

  2. Building tools to help accelerate the work as quickly as possible.

share|improve this answer

Automation is a good option in scrum or agile development only after some baselined workable product is rolled out typically after one of 2 sprints are over ,because till then requirements are quite fragile and attempting automation is not a good option, till then manual testing /acceptance tests at unit level, integration level based on either behavior driven or use case driven/model driven methodologies supporting TDD,do provide appropriate test coverage and defect identification. However once we have a workable product,we have some maturity in the test cases, they thought of as ideal and effective candidates for automation against which we can define priorities as defined by the product owner and start working further into building regression suites. so as the functionalities build up with subsequent sprints we get a more robust automation regression cycle coming up in shape and over all effectiveness and efficiency of the testing effort increases exponentially.

share|improve this answer

According to my view, "Scrum is the best agile approach. In scrum, each sprint produces an increment, which is a potentially releasable product. Here, each increment must satisfy all acceptance criteria and pass the different categories of tests. Regression tests are a tiresome activity, especially in an agile process, which is characterized by nonstop changes and frequent deliveries. If project ‘Cost’ affect the project then we can perform Regression manually. If not then automation is best to handle regression. Selenium, Microsoft Test Manager etc. are examples of tools that the team can use to automate regression tests. "

share|improve this answer

With long term projects (2 to 3 years already completed) and if there are 100's of old test cases, we are doing the following for 'Master' Regression and Iteration Regression

  1. Master Regression - would be a big set of test folders with merged test cases under each functonality and the new functionality or changed functionality will update existing test cases

  2. ITERATION Regression - After every iteration ends, we are doing a manual regression of the stories/test cases in the next iteration. We test all the functionality of the previous iterations. In the upcoming iterations we do the iteration + 1 testing.

I understand that this involves more effort every iteration, however, with projects that have releases every 2 months or so, and if there are only 3 iterations for dev/testing, then I find this kind of approach is recommended.

Additionally, we use the 4th iteration as a regression one.

share|improve this answer

protected by Community Jul 3 at 16:49

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.