How much automated testing should we do?

Question

I work in a software engineering department consisting of several development teams.

The department is mature in most agile methods except automated testing as we have old manual testers.

We have lots of legacy software which needs to be automatically tested and also we want to auto test new features as part of our definition of done.

My question is, what process would you suggest. This is what I have proposed:

1. Create user stories for all the legacy features which needs to have automated tests written and put them in the backlog

2. Include automated testing writing as part of new user stories' definition of done.

There is some resistance as people don't want to automate test all new features.

Any suggestions?

Answer 1

This greatly depends on your release cycle, true Agile teams should try to release as often as possible. An Agile release cycle could be from a couple of times per day to at-least once a month. Having a lot of manual testing effort before you can release will slow you down. Thus you will want enough automated testing coverage to be confident to release often without slow manual testing sessions. Try to automate as much as possible if you want to iterate as fast as possible with your users.

With a Legacy system I would write automated test coverage for all new features and indeed make it part of the definition of done. Also start writing an automated test for each defect found in production as these are the brittle parts. Then identify the high risk functionality and start writing test coverage for these parts also, maybe put this in the backlog. Slowly the whole application will have basic automated test coverage.

Do not create a test backlog for everything, just for what you envision you want to test on the short-term. This because the ammount of work might look like an never ending story. It is possible that the test suite will make you feel confidant even without all the possible test-cases.

Resistance is logical. Mostly because the return on investment seems low. See that you automate time consuming tests first. Also I suggest the older manual testers read the Agile testing book or do let them do the Certified Agile Tester training to let them get a better understanding of the importance of what build quality in means.

Keep in mind the test pyramid and decide wisely where you and how much effort you put in automated testing. Preferable more in unit-tests than in UI testing.

Answer 2

Your suggestion addresses how to add writing an automated test as a work item in your Agile process, but does not recognize that some tests should never be automated.

You need to understand and address your team's objections to automating everything. Some of those objections may be legitimate. For example, some automated tests may require so much maintenance that the benefit will not justify the expense.

The right answer is probably a combination of manual and automated testing.

I suggest this process. First, choose a small pilot project to help you evaluate testing frameworks and estimate levels of effort. After that, decide which manual tests would be the most valuable to automate, and prioritize that work by level of effort.

Decide on a way to measure the benefit of your automated tests, e.g. testing time saved. Measure how long you spend writing and updating your tests, too. At Google, if an automated test's benefit does not justify its cost, they delete the test. Pruning ineffective tests should be part of your process too.