I am working on a project to automate legacy code, and I'm looking for a good metric to track my team's progress and estimate how long it will take to complete the project.

My thought was to take all the automatable test cases we have identified, and use the percentage of these that have been automated as the main way to report progress. I also thought that since many of our automatable test cases are about the same size, we could use an average time to automate each test case to give us some idea how big the project is and how much time we will need to complete it.

The problem with this idea is that many of the test cases have multiple parameters. I don't want the metrics I use to measure progress and estimate time to give the wrong idea of how many legacy features are left to be automated or how much time we will need to finish the project.

For instance, assuming my project is a custom browser that supports multiple search engines. Should the search feature using each of these engines be considered a separate 'automatable' test case as from automation perspective each of these is just a parameter of may be a single test method?

So if in the test automation code I have a parameterized test, for tracking purposes of an Agile project, should I consider it to be many test cases as there are search engines?

What if I am just counting test cases and not automatable cases, then should I treat separately?

Also, should my test cases account for implementation differences? Let's say my app supports my browser and Chrome, should I account for Chrome test cases separately?

If there is an industry standard practice here, I would like to follow it.

My main concern is that I choose metrics that will give me results that are not distorted by being too large or too small.

  • Why do you count test-cases? What are you going todo with this number? What are you going to track and why? If you know why you will also know how to count. – Niels van Reijmersdal Jul 23 '17 at 6:02
  • To track progress of how much automation is dons. We are automating legacy code, that's why. I too prefer working code over doc but to understand how much of work is left and how we are progressing, I need the total count. – Paddy Jul 23 '17 at 6:10
  • So more testcases is more progress? I would look at a metric like code-coverage or test-coverage to find gaps in testing. – Niels van Reijmersdal Jul 23 '17 at 7:49
  • No I didn't mean that way. On the contrary I think that lesser the tests but focused the better. Hence the question itself. I want to ensure that adding parameterized variations of a test case isn't giving a bloated idea of having many cases and not giving me how many real features I am testing. Reason I mentioned progress is related to this concern - how many more legacy features are left to be automated and how much time it will take. Adding parameterized variations to the count makes it seem need even more time to achieve. – Paddy Jul 23 '17 at 11:03

There is no one correct answer

That said, I would look at it this way:

Assuming that your test code is SOLID and DRY, any parameters that have no impact on the flow through your test code don't count as "new" test cases. Parameters that change the flow through your test code do count as "new" test cases.

So, in the example of multiple search engine support, if your test case code looks something like the pseudocode/structured English below:

[TestParameters=[(engine = "Bing", search = "Object oriented code")(engine = "Google", search = "Object oriented code")]
public void FindFirstObjectOrientedCodeHit()
  ExpectedResult = "" //URL of expected first hit
  browser = LaunchBrowser(); //defined elsewhere to launch your custom browser
  result = SearchWithEngine(browser, TestParameters(engine), TestParameters(search)); // defined elsewhere
  Assert.That(result == ExpectedResult);

that would be one test case. You could also include expected first hit as part of the parameters.

If on the other hand you have to do extra processing or have a different flow through the system, I would define that as a separate test case for each flow, so if search engine X requires a login, there would be a test case FindFirstObjectOrientedCodeHitAfterLogin() which would be separate from the normal search test case.

| improve this answer | |

As you say you are doing an Agile project, take this value of the manifesto into account:

Working software over comprehensive documentation

Sounds to me you are making comprehensive documentation and are searching for confirmation that you are on the right track.

Question yourself, do I really need to know the number of test-cases? I like to see the number of tests grow as features are added, but I couldn't care less for an exact number. But if you have a case where you need the numbers by all means record them. Just be sure you are recording them for a clear reason and that afterwards you are using them. Don't create numbers and metric, statistics just for reporting sake. It is just a waste of time.

The industry standard in Agile is that you record what you need for your project. With a preference for recording the least as possible.

| improve this answer | |

This sounds like data driven testing to me.

To count the cases you need to calculate the total workflow routes.

For example, if you have three test cases: A, B C and, based on varying data, you have 5 sets of possible workflows for test A, 4 workflows for test B and 12 workflows for test C then you have 5 + 4 + 12 = 21 tests.

In terms of progress, when tests A and B are done you are 9/21 or 42% (as opposed to 67%) done.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.