I am working on a project to retrofit "automation" to a legacy project, it's a tricky beast, people just want to know what the progress is on it.
Is there any real way to measure automation progress, other than observing issues caught ?
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.Sign up to join this community
For a legacy system, particularly when there isn't any lower level automation to leverage, I find it helps to start with some analysis to determine first the major functional areas of the application and how much use they receive; then repeat that analysis for sub-sections within the major areas.
This allows me to have a basic guide to approximately how much is left, and a way to ensure the most important parts of the system are covered first.
As an example, say your application is an integrated point of sale and inventory management application.
Major features would include:
Login, purchasing, and pre-order would be your highest priority modules to automate, with reporting as your next priority.
Within the on-site purchase module (just to take an example) you might have such functions as:
And so on.
Since each application is different the user base and use profile will differ per application, so you need to do this for yourself.
Knowing what features get the most use also helps to schedule automation for new functionality - depending on how likely it is to be used, it might be necessary to prioritize automating a new feature over catching up on an existing feature.
One approach is to spend some time creating a master list of all the application functionality. The simplest approach to that may be to base the list on UI functionality. A more complicated approach is to focus on the back end, based on code, classes, methods, etc.
One you have the list - say 278 total functions then you can measure your progress in automating them, e.g. when 28 are done you are 10% through the list.
This can be done very simple with just the master list and how many are done. It is not precise however as there are many factors such as shared functionality code, different function complexity, frequency of use, data driven testing, validation verification's, etc. However it is usually better than nothing and in an Agile environment can be refined and adjusted over time.
TL;DR: Code coverage
retrofit "automation" to a legacy project
What does legacy mean?
To me, legacy code is simply code without tests.
The Legacy Code Dilemma: When we change code, we should tests in place. To put tests in place, we often have to change code.
-- Michael Feathers (Working effectively with legacy code)
Michael Feathers describes some good methods how to write tests around un-testable code. Now you should refactor the classes, so you can write unit-tests them. Now you can remove most of the larger end-to-end tests again.
The retrofit word scares me as team might fall in the testing ice-cream cone trap. You will want to aim for a good balance between unit, integration and end-to-end tests. Here I think code coverage makes sense as measurement, because it shows you which branches you have not yet covered by a test. Once you have all branches covered you can start with the refactoring probably.
How do you know if your code is covered by tests? Code coverage. If you ask me I would pick branch coverage as your main metric to measure progress.