4

There is web application product which is tested by test automation (selenium webdriver based test project in scope of the whole project) Client is asking what is the test coverage by automation of our product?

The question is what is the best way to track, to make some docs, etc for coverage metrics, that can be used not only by client, but also by developers if they want, for example - to launch only some corresponding bunch of test scenarios based on that metric coverage ?

Improve this question
1
  • 1
    What kind of automation coverage do the client and developers want? It's relatively trivial to have a high level of code coverage and miss critical paths through the system. It's also usually impossible to test all possible paths through most software. Knowing why you want the information will help us give you better answers.
    – Kate Paulk
    Commented Jul 17, 2014 at 16:28

2 Answers 2

Reset to default
4

Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!"

When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when the test suite is run. However, you should check out this StackOverflow answer on code coverage as a metric (it's not as simple as it sounds): https://stackoverflow.com/a/90021/471206

You're not looking at unit tests. You're looking at functional automation, at UI level. You could aim to cover every line of code via the UI, but that would be pointless, wasteful and slooooooooooooooooowwwwwww. At that level, you should probably be thinking about ... well, functionality. And there are quite a lot of different ways you could choose to model that. How will depend on what your client is really asking for.

There's an excellent pair of articles that explain why coverage isn't so simple: Got You Covered and Cover or Discover.

What I would suggest, is that you report several things to your client:

Your coverage against unit and integration tests - though I would explain that 100% coverage isn't desirable, so that they understand you're aiming to test the important stuff.

And your coverage against functional level tests, against the model you've chosen to use. That could be that you've identified the main workflows in the app, and have chosen to use your UI test suite to cover those. It could be some other model. Without a bit more information, it's hard to suggest what might be a good one (and that's also dependent on what's important to your client, so you do need to have a conversation with them too).

Either way, you'll need to explain why you chose that model, and why you think the results you achieved are a good balance of cost vs value.

Improve this answer
1
  • Note: the reason I am not talking about number of test cases is that I believe that's a meaningless metric. Show me any test suite, and I bet I can add tests to it. Talking about 100% of your test cases being covered is begging the question of why you picked those tests and not some other set from the infinite possibilities available. If you're already reporting numbers of test cases to your client, you might think of this as an opportunity to provide a slightly more nuanced report.
    – testerab
    Commented Jul 23, 2014 at 6:55
2

I would give them a percentage number of test cases automated of all test cases that can be automated. The key here is to develop the full list of test cases that "can" be automated. What test case management tool do you use?

Improve this answer
1
  • I vote for the answer above. (Test cases Automated/Total Testases)%
    – Web Nash
    Commented Jul 22, 2014 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.