3

Just want to give an example: We have regression test suite of 1000 TC. Currently it take 10 people to execute it manually in 10 days. After automation of this suite, lets assume we will need 3 people to execute.

If we have a major functionality change which affects 500 of my test cases, we will need 7 people to fix scripts and re-run and update. i.e. basically maintenance of scripts. Essentially 3 fr execution and 7 for fixing.

Where do I see saving from automation of my regression test suite

Assumption: existing scripts work perfectly on current codeline, have followed Selenium Robot framework.

2
4

This is a pretty complex one to work through, but you can do it with some data diving as well as well as the information you've given.

  • Time and Labor in your group - assuming you have your automation suite in place, each time it runs, you are running (assuming that the suite runs in under a day) with 3 person-days for a total of 24 hours. The cost to run manual regression is, from your post, 100 person-days (10 people x 10 days) or 800 hours. So each run without modification saves 776 hours.
  • Time and Labor external to your group - This will take some data diving. You will need to work out how many regression issues, on average, escape to production per release, and how much time is spent dealing with them post-release. Once you've got that number, you can compare against the number of regression issues, on average, that escape after your automation is implemented, and figure out a time saving. If you've targeted your regression automation properly, it could be a substantial amount.
  • Regression bugs not released to production - you'll need to do some digging here, too - this time to locate all the regression issues that are caught by your automation and work out how both how likely it was that the issue would have been caught in manual regression, and how much time was saved. If you're running the automated regression on a daily or more frequent basis, you should see a lot of savings here, via an average time per issue.

To use some numbers pulled from air: - Every run of your automation saves 776 hours. - If it takes on average 500 hours to update the automation per deployment, then your time savings are (776 * number of runs per deployment) - 500. If you run the automation 10 times per deployment that's still over 7000 person-hours available for other tasks.

If you design your automation well enough, you should be able to reduce your refactoring time significantly, increasing your time savings.

And if your management requires a money amount on estimated savings, you'll need to include the development time to create the automation as a sunk cost (because it will happen no matter what kind of automation is used). Say, 5 times the time cost of a full manual regression run to do it properly. That means that by the sixth run of your automation, you'll be into positive ROI - but as you implement automated runs on a smaller scale, you will start seeing benefits).

2

Saving time as Kate Paulk mentioned is not the only benefit, and wrong way to look at the problem, IMHO.

Automation increases quality and stability of your releases. Allows to refactor parts of the system and be sure nothing important is broken, if all test pass.

Also, increases the job satisfaction of the testers (less boring repetitive condition-checking, more interesting investigative testing), so lowers turnover. And more.

4
  • 1
    I think the bigger question is how to quantify those things. If you work in an organization that makes decisions based on measurements, it's not enough to claim automation has intangible benefits. – user246 Apr 26 '17 at 18:53
  • Sadly true - sometimes you have to show verifiable numbers to those who want numbers. Time saved within the team is one of the easier numbers to use. Time saved by bugs caught earlier is a little harder, but helpful - and time can be easily converted to money for managers who want that. – Kate Paulk Apr 26 '17 at 19:54
  • 1
    How do you measure quality of releases? Refactoring? If you care only about time saved, people will NOT refactor. And may leave to better managed companies, where work is more satisfying. And velocity will slowly decrease. Be careful what you measure: because that's what you will get! – Peter M. - stands for Monica Apr 26 '17 at 21:38
  • 1
    I know, @PeterMasiar, I know. If I'm measuring release quality, I'm looking for the number of customer-reported issues that are real issues and not feature requests disguised as issues - and reporting proportion of regressions vs new issues. If you're doing well, your customers won't refer to your releases as being "safe" or "not safe". – Kate Paulk May 1 '17 at 11:31
1

I've seen nice graphs for this kind of thing, where the y-axis is cost, and the x-axis is how many times the tests are run, and two lines, one for manual and one for automated.

The manual one just has a fixed slope since there's a fixed cost for running the tests manually (10 engineers * 10 days * daily wages).

You can show a similar trend for automation, showing the large sunk cost (automation engineer * time * wages would be the starting point, even before any test cases are run), and then a line with 3/10th the slope since you'll have 3 engineers executing the tests.

You can then see the savings where those two lines cross . . .

Adding in refactors and things to such a graph is left as an exercise for the reader.

0

((Hours spent on automation including writing/execution) / (manual test creation/mod work) + (manual test execution time spent on tests)) * (test coverage percentage) = % saved by automation instead of manual work

Example: 40hrs(a) / 20hrs(edit work) + 40hrs(execution) = .6667 * .3(30% coverage) = .2 (20% overall increase in efficiency by only automating 30%).

You then multiply this number by the fact over time, so say the initial work is 20% improvement, but then repeat efforts decrease the automation hour timeframe thus improving the % improvement drastically over time. I'm used to seeing a negative result on initial start as sometimes the upfront work on automation takes longer, but the follow on efforts should more than make up for the improvement and you can show the trend over time of improved efficiency.

If you don't test some things in regression then you can add it as "additional coverage" that isn't possible manually.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.