4

I work in Product based company as a QA/Test Engineer. For ex:I have been assigned a new project/enhancement that would be delivered to the clients in the next production release. The Business Analyst publishes the "Functional specification(SRS)' of the new feature/modules. Next, i start writing Test cases/scenarios for that.

After that the DEV team releases the build and i start doing the "functional" testing. I encounter issues, report them, it gets fixed, i re-test them and this goes on until i say that all is OK. The product is finally released to a production server.

Now, if management asks me about Test coverage, Test Cases coverage, then what is the best answer. My team does not follow Agile. It still follows the old waterfall based approach. There are no sprints or user stories.

Now, i believe that "Test cases are not Testing" and industry is moving towards a performance culture. Now, skilled Exploratory Testing is more important than Test Cases metrics. Then why should number of Test cases written define Test coverage. I wrote 100 test cases, which covers all of the requirements in the SRS. Then the question is, that is my coverage 100%. Why is this kind of coverage measurement wrong and outdated?

Improve this question
3
  • I think you're confusing code coverage with test coverage - the first being a whitebox technique which measures execution against code, the second being a blackbox technique which measures execution against the test plan. So, to keep it simple, say you've written 100 test cases and executed 30 of those... then your test coverage (and what you report to the business) is 30%.
    – dvniel ♦
    Commented Mar 20, 2017 at 15:11
  • 1
    If they're asking whether you've covered all the functional changes, then it's a Yes or No answer, because you've written your tests against the requirements delivered by the Business Analyst. If the Functional Specification has described a functional change, then you've covered it in your test plan.
    – dvniel ♦
    Commented Mar 20, 2017 at 15:12
  • Test coverage is actually requested and it's not the same thing. Especially being black box there is no code coverage capability since the code and underlying components and functionality is specifically hidden from the tester. This is often the case with a vendor customized or third party customized product. I would always recommend white box testing as well in that strictly black box only testing would leave unknown gaps which could lead to issues in the future.
    – mutt
    Commented Mar 20, 2017 at 15:54

2 Answers 2

Reset to default
4

Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code and components.

Therefore when you are prepping for black box testing you should be coming from the user perspective who utilize the application. This is any and every user standpoint including super users of the application, but not a developer view point. Ignore code coverage and focus specifically on the usage situations from the user perspective. Once you get there then build onto it by adding in boundary/edge case testing as well. Always top it off with thorough exploratory testing which may bring up different combinations that weren't tried during standard and boundary testing.

Be thorough with the below:

  1. User perspective to looking at the application
  2. Boundary/Edge cases from the user ability/perspective
  3. Exploratory testing to test various combinations

If you have the above fully covered then you can say you have solid test coverage from a black box testing approach. The key is functionality coverage with the 3 items, so if you have 45 pieces of functionality and you have all of the above for all functionality you can say 100%. If you have some of the above but not all, then you can come up with an equation to provide management on test coverage execution. (i.e. 38 has all 3 and 7 has 1 that is ((38*3)+(7*1)) / (45*3) = 89.6% coverage) You can tweak it based on risk and give more or less value to certain parts as needed.

Improve this answer
0

In highly regulated industries no changes get done without a formal specification. This is done for reasons of liability and certification. If you're designing software to keep people safe you damn well be sure that you are covering every contingency.

What your manager is asking when they are talking about test case coverage is "Do your tests cover the spec?".

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.