Is manual regression testing an technical debt for agile teams aiming for continuous delivery?

Question

Background: Following are not my assumptions or expectations but in my last couple of interactions with different project teams, I am seeing these patterns where it is expected (from testing by the overall project team):

Assumptions:

1).Test Coverage:Where all or most of testing are aimed(by project team) to be covered in either unit tests or automated user acceptance tests.

2).No automation spillover between sprints:In each sprint, all testing for that sprint should be completed in that very same sprint and no spillovers expected in terms of what is still left as manual testing activity once the sprint ends.

3).Full automated Regression Testing: From regression testing perspective, there should not be any accumulated leftover manual testing from each sprint which is still being done manually.

4).Product Quality & Continuous Delivery: As the team aiming for Continuous delivery, the overall expected product quality is very high where the product can/will be delivered anytime to production.

Technical Debt: And if we still have something which we are testing manually after few iterations then it is considered as a technical debt of testing by the project team standards. What I am trying to understand here is: is it a reasonable/right expectation of testing in general?

which brings me to the overall larger question:

Question: Once "Manual Testing" starts becoming a technical debt in a agile team, and where team is highly focused working in short, tight (no spillover), fast sprints and aiming to achieve continuous delivery, what can the team do to reduce it over a period of time?

Expectation: I am looking for the valuable thoughts/experiences of fellow QA folks, who have been in similar situations and particularly what worked for them ultimately.

Disclaimer: These are not my thoughts but my perception of the expectations from a couple of different project teams.

Answer 1

In my experience, when the lack of automated regression in a delivery pipeline becomes technical debt, you have exactly one option.

Treat automated regression tests as code and sprint on them the same way you sprint on new features and bug fixes.

What that comes down to is that if your team(s) didn't set this up at the start, you will have to either waste time retesting things you shouldn't need to retest or spend extra catch-up time building the automated regression in.

Since even in a continuous integration pipeline it's difficult to build UI automation (to be avoided as much as possible) or high level integration tests (for API end points) until the feature is coded, it's not uncommon to schedule automation of a feature to run a sprint behind development of that feature - but it still needs to be part of the scheduling.

My suggestion would be as follows:

1. Identify test areas that are automation candidates during manual testing. These become the automated regression user stories for the next sprint.
2. Identify tests that should not be automated. These are going to be the core of a lightweight manual regression that should be run only if that part of the system is changed.
3. Until automated regression is running no more than 2 sprints behind initial development, all team members spend between 1/2 and 2/3 their sprint time on catching up the automation (you can take it to 100% but this means no new development or bug fixes).
4. Once automated regression is caught up, between 1/3 and 1/2 the sprint time goes to maintaining and adding new features to the automated regression.

This is of course the ideal world. I've had to struggle to get a tiny fraction of this in every testing position I've held - which is where my final piece of advice comes from: If you don't take the time to do it right, you will be forced to take the time to do it over.

Answer 2

There are an awful lot of variables to consider here. But here are some points all the same.

Manual Testing vs Automated Testing This point alone is probably worthy of a book and is one of the most frequent questions I see. E.g. I'm new to a project, what should I automate? Things to consider here are...

• Does take a long time to execute the manual test?
• Do I need to run this during every sprint?
• Is it viable to automate? E.g. technically possible. ROI in terms of time to automate and time to run manually.

If the test is isolated or a small bug-fix test, then I might consider not running it every sprint. There's probably little value in automating it too.

If for example, we're testing a login screen which uses an API, then you'd think the manual tests to check as many eventualities as possible would be very time-consuming. On the plus side, it's technically straightforward to automate. Once the basic test is automated, you can then consider turning this into a data driven test whereby your inputs and expected results would be a row in the spreadsheet. This enables to test 1000's of combinations in very little time. Plus, you can add further combinations in later sprints without having to modify the test itself.

Reducing Manual Testing On previous agile projects, I would devise manual tests for each ticket assigned to me. At the end of the sprint, I would then look through the manual tests to see what is worth automating. I'd then automate and run my manual and automated tests in the next sprint as regression. New items in this sprint would be tested manually, and then considered for automation at the end of that sprint.

So, new stuff, I'd manually test in that sprint. For old stuff, I run the automated and manual tests as regression. This results in the regression suite growing on a sprint by sprint basis. Hopefully, there'd be enough automation opportunities that would save me test time to focus on those tests that I think need to remain as manual tests.

In a lot of posts I see on here, it's almost as though people think manual tests are BAD and automated tests are GOOD. IMHO, a good set of tests can and will probably include both. Don't get obsessed with trying to automate everything to eliminate your manual tests. Manual tests are still very valuable.

Re we still have something which we are testing manually after few iterations then it is considered as a technical debt of testing by the project team standards My opinion here is that technical dept is more applicable to maintaining automated tests that are probably not robust enough for your needs. Your automated tests should be reliable and robust. I don't see manual tests as technical debt. You may find yourself in a position where your manual tests are taking far too long to execute in a sprint, but I wouldn't ordinarily class them as a technical debt.

EDIT - Based on Niel's comments and the link to Technical Debt Quadrants, I can see how manual tests may incur technical debt. That said, I'd still be wary of attempting to automate everything and seeing any manual tests as technical debt. My approach would still be the same, automate those things that there is a benefit in automating. Don't automate manual tests that may be too complex or time-consuming versus running the manual test.

To sum up:-

• Look for automation opportunities, but don't try to automate everything.
• Manual tests have a place in your tests Review your tests frequently to ensure you're not running needless tests.
• Don't attempt to use a single tool for all tests. Use a variety of tools instead. I mean, you wouldn't use a screwdriver on a nail, you'd use a hammer.

Answer 3

It depends. Part of me is concerned with the tone of the question, to be honest. I'd be concerned that automated testing is being done to the detriment of manual testing, although some types of software would probably do OK with only automated testing.

Yes, you absolutely should automate that which you can. And some types of software development, and testing, are more suited for automation than other types.

However, the whole point of automation isn't the automation, it's to free up people to do things that only people can do, and some types of testing can only be done by people. The further you get up the complexity stack, the harder things are going to be to automate, or to automatically determine the results of the testing. If you're testing something that doesn't have much interaction with other components/products, you may be able to automate a lot of testing. But the issue with automated testing is that it's very easy to groove your testing.

I work in a field where there's a huge amount of automation, and it's great, as long as nothing unexpected/unplanned for happens. When something that wasn't planned for happens, you can have problems, because, well, who knows what will happen? The automation may do the wrong thing, the automation may not notice the problem . . .

Anyway, the point is that, potentially, manual testing is technical debt. But automated testing for the sake of automated testing is also technical debt. Only automate things that can safely be automated, and don't overlook things that can't be automated, but should still be tested.

Answer 4

Yes, I think you can group lack of test automation under technical debt.

We don't have time for test automation, so we do it manual

This sounds like "Deliberate" and "Reckless" on the Technical Debt Quadrants.

Worse would be "We don't know how to automate this testing", which is "Inadvertent" and "Reckless". Go get some skills ;-)

We know we should automated our manual test scripts

This would be "Prudent" and "Deliberate" and it sounds like technical debt. You know you postponed it, maybe for a valid reason, like we had to ship it to get feedback. Now you have to repay your debt.

If it is a strategic goal to practise Continuous Delivery everyone in the project should be aware you are gathering debt. Make it visual, either in the backlog or in the hallway. Like hours spent manual testing on each release.

Update:

Preventing technical debt:

1. Make the organisation (Product Owner, Management) aware that debt is being created.
2. Add tech-debt to the backlog, start with defining it and removing it every iteration.
3. Add tech-debt prevention practises to your definition of done (e.g. create automated e2e coverage, etc)

Answer 5

My take on what you are saying is:

• too busy to automate
• too hard to automate within the sprint
• insufficient belief / attention / people to do automation.

My approach would be:

1. Put more effort into convincing people of the value of testing. This is a massive topic so I can't dive into tons of detail here.

2. Time your testing. Do 1 pass of really thorough, intense testing of all the combinations you'd like to test and find how long it takes. Now do the basic math: how log it takes X how often you do it... per year. Once you have this figure you can make the case of how much time and money you will save. You will need to include the 'take time to write the automation' in this estimate. Recognize that some testing can take longer than the app code. In some cases much longer.

3. Push for smaller stories and break things down more so that you can include testing with the app code. This is not always easy but with effort I have seen it be successful.

Answer 6

As you are in a situation where continuous delivery is used, it is imperative, that every step is automated, in testing or otherwise. When there is a manual step you do not have CD.

You need to include the testing (automation) effort in your sprint planning. If automated tests cannot be completed in a sprint, the scope should be reduced.

Every manual effort is expended, every automated effort, on the other hand has an ROI and is an investment.

As you are agile, you should follow "Individuals and interactions over processes and tools" and make sure, your team discusses these issues in each sprint and acts accordingly.

Answer 7

I don't know what it means for a sprint to be "tight" but I assume that is supposed to be a good thing.

The question asks about going fast but does not mention anything about quality. If all you care about is going fast, and if an automated test lets you go faster than a manual test, then manual testing is an impediment to going blazingly fast. Whether or not you label manual testing as technical debt is pointless.

Of course you might care about more than going fast. You might also care about producing software that's reliable, meets its performance goals, and is usable. You might also care about spending money in an effective way. Sometimes the best way to meet those goals may be to do some manual testing.

Answer 8

I worked on a team recently where a significant amount of the manual tester's time was spent writing test cases based on requirements. The requirements were in a BDD format, but the test cases were in more of 'traditional'/imperative format (ie. 1.login 2.go to the data object page by clicking on the link on the left . .. ) there was a significant amount of time spent just keeping this stuff organized, or dealing with the lack of organization. This, I felt, was time that could have been spent just automating the requirements directly. That's kinda the point of BDD, after all.

There's also going to be moments where the engineer can test something manually before a release, and then never test that particular change again, such as tiny bugs or removed features. There's no reason to save those test cases with the list of stuff that may or may not be automated.

Another strategy is to make sure that the tests (automated and manual) focus on high priority features. The end users should be able to login, for example, but there may be features that are primarily used by internal users that don't have to be tested all time.

IMO, software development is getting faster, more reliable, and users are bit more forgiving of wee little bugs if they love the product. Features, once written and deployed to production, don't suddenly sprout bugs from other feature changes like they used to. I would encourage manual test engineers (or engineers who are doing manual testing before automating) to focus efforts on 'what is the product manager and developer forgetting that is going to inadvertently destroy us?','What should be a priority for on-going automated regression?', and 'What might have to change in our current regression to keep it up-to-date with our requirements?' The goal is to get to the point where you can trust your regression suite to come back with failures that are actual, deploy-delaying bugs, so your greatest manual QA efforts are focused on the absolute latest features and releases.

You might find that 'breadth not depth' mantra can help with that. User Acceptance Tests should absolutely cover Create Read Edit Delete and Login. They don't have to cover 'header is blue and big, hint is small and grey.' The unit tests should cover the 100 permutations of stuff we can put in this input, and your user acceptance tests should use selenium to make sure Mr. Graybeard in the corner office can load the page and print his TPS report. They tend to run slower, but when Bob calls about his missing report, Automated UAT is great for figuring out where the feature went wrong.