How to approach testing of a mature application

Question

What should someone prioritize when deciding to add automated tests for an application that has already been in the market for a long time(3-4 years) though it still receives updates? Assume that most of the obvious bugs have already been found and the GUI shouldn't change other than new additions.

E.g., assume that this application is Calculator with a login functionality that if used, allows you access to scientific and financial views. Normally, among others, I would test that drop down menus have all the options available, buttons are of same size and at the correct place, login functionality, the scientific/financial views are available only after login and of course the core functionality which is what happens when each button is pressed. New updates could include a new view(e.g. programming) or additional buttons in existing views.

Which of those tests should I still automate?

Relevant book, article proposals, your experience are all welcome.

Edit: added the calculator example.

Answer 1

The vast majority of automated testing isn't done to uncover new bugs. It's done to provide some assurance that updates don't change the expected behavior of the application. The most common question asked by automated tests is "Did these new changes break some existing functionality". This sort of testing won't root out non-obvious bugs, but will detect whether bugs were introduced during new development.

There are many trade offs, when determining which tests to prioritize automating. Some of these considerations are; the likelihood that a feature will break, the cost associated with not catching a broken feature before release, the ease of testing the feature manually, the difficulty with automating tests of the feature, and the cost of maintaining the automated tests through future development. While it is informed by the technology, this is entirely a business decision.

The definition of quality changes between industries. What I consider good enough to ship when working on homework assignments is very different than what I'd consider good enough to ship were I to be developing software for a medical device, or airplane.

Answer 2

For this kind of "old" application you are not probably targeting to validate new functionality with automated checks, but to get confidence with application after new release. So, regression tests come to mind, because new functionality is probably tested before deployment by someone.

As Joao wrote, for best results you should think in terms of risks and values for interested groups.

So, ask yourself or product owner, clients, or colleagues who worked with the application previously:

• What is most important for user? What they use most? This is good for gain at least some certainty about the application and that core functionalities work as user expect.

• What do you test repeatedly before releasing a new version? This is useful for not wasting time with something which you already repeat. You will have more time to explore your application in new ways

• Are there any time-consuming operations? Are they important and can be reasonably automated checks for them?

• What is already tested on some other level? For example, as unit test created by developers. Or perhaps by other departments. It there is good suite of unit test developed by developers, then you can maybe assume that some basic thing are tested by this way and you can omit them to certain level.

• Do bugs come back?

Create a list of answers to questions and perhaps at that moment a priority of tests will be more clearer.

There is a myriad of questions, but they depend on context.

Probably do not expect to find significant number of bugs with automation, but expect to find some bugs during development of automated checks.

I just found 1.5 Million lines of code. 0 tests. Where should we start? maybe that post can provide you some more clues