What should someone prioritize when deciding to add automated tests for an application that has already been in the market for a long time(3-4 years) though it still receives updates? Assume that most of the obvious bugs have already been found and the GUI shouldn't change other than new additions.

E.g., assume that this application is Calculator with a login functionality that if used, allows you access to scientific and financial views. Normally, among others, I would test that drop down menus have all the options available, buttons are of same size and at the correct place, login functionality, the scientific/financial views are available only after login and of course the core functionality which is what happens when each button is pressed. New updates could include a new view(e.g. programming) or additional buttons in existing views.

Which of those tests should I still automate?

Relevant book, article proposals, your experience are all welcome.

Edit: added the calculator example.

  • Clearly, as with applications in any situation, risk-based prioritization is the path; but for more details, I would say more information is necessary. "GUI shouldn't change other than new additions" implies only that frontend may be considered low-risk, but a manifold of other risks can exist. – João Farias Feb 20 at 18:32
  • I added an example so that there is something to work on – vtheodor Feb 21 at 11:03

The vast majority of automated testing isn't done to uncover new bugs. It's done to provide some assurance that updates don't change the expected behavior of the application. The most common question asked by automated tests is "Did these new changes break some existing functionality". This sort of testing won't root out non-obvious bugs, but will detect whether bugs were introduced during new development.

There are many trade offs, when determining which tests to prioritize automating. Some of these considerations are; the likelihood that a feature will break, the cost associated with not catching a broken feature before release, the ease of testing the feature manually, the difficulty with automating tests of the feature, and the cost of maintaining the automated tests through future development. While it is informed by the technology, this is entirely a business decision.

The definition of quality changes between industries. What I consider good enough to ship when working on homework assignments is very different than what I'd consider good enough to ship were I to be developing software for a medical device, or airplane.

  • Your answer was too generic about automation testing so I added an example so that you can be more specific if possible. – vtheodor Feb 21 at 10:11
  • @vtheodor What application you're testing is only part of the story. If I was writing a calculator as an internal tool on my lunch break to make my teammate's job easier, my definition of quality would be very different than if the calculator was being used to determine safe dosages of radiation. Failures in a quick side project may leave a coworker slightly miffed. Failures in a safety critical application could hold me liable for millions of dollars in damages. Ask yourself what is most likely to break, and how bad will it be if it breaks? Use that to determine where your risk lies. – sphennings Feb 21 at 16:25

For this kind of "old" application you are not probably targeting to validate new functionality with automated checks, but to get confidence with application after new release. So, regression tests come to mind, because new functionality is probably tested before deployment by someone.

As Joao wrote, for best results you should think in terms of risks and values for interested groups.

So, ask yourself or product owner, clients, or colleagues who worked with the application previously:

  • What is most important for user? What they use most? This is good for gain at least some certainty about the application and that core functionalities work as user expect.

  • What do you test repeatedly before releasing a new version? This is useful for not wasting time with something which you already repeat. You will have more time to explore your application in new ways

  • Are there any time-consuming operations? Are they important and can be reasonably automated checks for them?

  • What is already tested on some other level? For example, as unit test created by developers. Or perhaps by other departments. It there is good suite of unit test developed by developers, then you can maybe assume that some basic thing are tested by this way and you can omit them to certain level.

  • Do bugs come back?

Create a list of answers to questions and perhaps at that moment a priority of tests will be more clearer.

There is a myriad of questions, but they depend on context.

Probably do not expect to find significant number of bugs with automation, but expect to find some bugs during development of automated checks.

I just found 1.5 Million lines of code. 0 tests. Where should we start? maybe that post can provide you some more clues

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.