Quantifying the value of Automation

Question

I’ve just spent the last 6 months replacing a couple of old frameworks one using enterprise toolsets with a shiny new appium/webdriver framework. Everything has gone well with good buy in from both the testers and management. Our test runs which previously were taking 40 hours + are now far more manageable. We are in the process of planning for the next year and determining the milestones for the Test Engineering team. Management has seen the value in reducing the run time of the nightly run, using open source tools and creating an easier to use framework, but they have asked me now to provide them with metrics or the value that will come from some of the tools/processes that I've seen implemented at other companies.

This might not be the best place to ask all of these, but I’m hoping people here might have gone through something similar. Does anyone have any way of valuing:

• Comparison reports/run history with a combined log file from multiple sources to enable testers/devs to faster diagnose failures
• Running tests more regularly/on checkin rather than just nightly, or running just the affected tests
• Reducing flaky tests or rerunning flaky tests
• Bots to automatically segment tests and raise defects
• More regular security/performance testing rather than at release
• Star increases for apps
• Code coverage stats
• Dashboards

Cheers, James

Answer 1

Yep, been through a lot of this myself. In general, reliable automation with failures that are easy to diagnose can dramatically help increase velocity and confidence and save money. I typically try to create automation that is part of a CI pipeline with the following 2 goals:

1. Ensure I cover at least the happy path of every feature to get immediate feedback on the quality of all features rather than having to wait for a manual regression which could not uncover issues for days or weeks.
2. Cover as close to 100% of all regression tests for specific features to reduce the overall cost and time of regression testing.

To specifically address your points:

• Comparison reports/run history with a combined log file from multiple sources to enable testers/devs to faster diagnose failures

  • Enabling your team to faster diagnose failures is a key to ensuring that the automation adds value. Taking more time to diagnose failures means that you need a longer window to run tests and look through results, you can measure this in how much you pay your team members for X hours of additional investigation, as well as loss in time to market for longer feedback loops.

• Running tests more regularly/on checkin rather than just nightly, or running just the affected tests

  • This directly impacts how quickly you become aware of issues, and helps you pinpoint the exact check-in that caused the problem. You can measure this in terms of hours spent on the development team trying to determine which check-in caused a defect.
• Reducing flaky tests or rerunning flaky tests
  • Re-running flaky tests can actually mask intermittent issues that are defects. I usually caution against that. If you are talking about flaky tests that you KNOW are due to automation issues, you should fix them, not re-run them. Taking time to address these reduces noise, builds confidence in the automated results and saves time that you would have been spending determining whether there was actually a defect in the product or not. You can measure this in terms of hours spent investigating flaky test failures - or worse, ignoring those failures and letting a real defect slip through.
• Bots to automatically segment tests and raise defects
  • I've got mixed feelings on this one. I like the idea of segmenting tests automatically, like by evaluating the stack trace and finding similar failures that could be due to the same root cause, but automatically raising defects can be a bit tricky to get right and I typically do not do that. Measuring the value of segmenting tests can again be put into hours spent investigating failures. If you have 30 failures in one night and each takes 10 minutes to determine what happened, but you can group 20 of them immediately back to 1 root cause then you just saved over 3 hours of investigation.
• More regular security/performance testing rather than at release
  • This one seems like a no brainer. 1 single security breach can spell disaster for a company. Up to and including a mass exodus of customers and bankruptcy. Bad performance or memory leaks that cause the app to freeze up or crash similarly can have huge impacts. Even if you avoid one fire-drill of pulling people into the office in the middle of the night, this one pays for itself.
• Star increases for apps
  • Not sure what you mean by this.
• Code coverage stats
  • I love using code coverage stats for my automation. More than anything, it helps show me where I should be focusing my testing and what areas are not covered. It also helps me determine which tests are truly equivalent and cut down on the ones that are duplicates. You can measure this in terms of reduction of defects due to covering more code paths.
• Dashboards
  • These are typically more useful for management, communicating status. You can measure this in terms of less time manually communicating status to management.