How to report test progress when you have blocking issues?

Question

I have a data migration project going on. There's a semantic mapping tool in the middle to translate between two DBs. I'm testing that.

However, some issues that I find are fairly...blocking. E.g, one that causes the wrong number of values to appear in every column in the new database. Fairly big.

As a result, any other tests I write that get masked by this issue I can't know if the test is right yet, or if once the blocking issue is fixed we may see more issues, or whether everything will pass.

The project manager wants to know progress and when I said it's tricky to tell because of this massively blocking issue, says "put it this way, how many tests have you written?".

To me this is a horribly inaccurate measure, as the tests could be wrong, or new bugs could cause new tests to need to be written, or we may run into even further blockers, let alone the fact that the tests will need to be re-run.

He finally insisted that I tell him how many tests I've written, at which I could say "80%, but..." and he said "well there's a metric!". I tried pointing out why it's not a realistic one to measure progress though.

Are there more legitimate ways to report progress to project managers when you have big blocking issues potentially masking further problems?

Answer 1

You need to meet your project manager half-way. Using an unrealistic metric does not help either of you, and you are right to be concerned about that. At the same time, you need to avoid describing your problem in so much detail that your project manager will not understand it. Given what you wrote, I think I would tell the project manager something like, "If you want an accurate status that you can measure against your schedule, you need to break down my work into N parts (where N is preferably a very small number): design, code, debug, etc. Here is where I am against those N parts. And here are some risks to my continuing to make progress."

If you already did that, and you are not satisfied with their response, you should amend your question with that information.

Answer 2

The metrics can be divided so that there is reportable status. 1) List of test conditions (i.e., test cases). Do you believe that you have a complete list written down? If so there is one metric (identification of the test conditions). 2) From the list of test cases, how many are completely fleshed out as far as what needs to be done to run the test? Your blocking issue probably prevents documentation of adequate detail on many of the cases you've identified. This would be a second metric (development of test case steps). 3) Test execution would be a 3rd metric. What %, if any, have been executed and passed. If there are none that wouldn't need to be re-executed after the blocking issue is fixed, this part is 0% complete. If there are some tests which aren't impacted by the blocking issue, you "may" be able to count those as completed.

Answer 3

Project managers want to track the overall progress by a % complete number. Here is a way which works (depends on your organization).

You could provide a report that goes something like this:

Total Tests Written - #count
Total Tests Executed / Tried - #count
Total Tests Passed / Found to be Working - #count
Total Tests yet to be Checked / Executed - #count
Total Tests that cannot be Checked - #count

In this way the PM can find the % that if found to be truly working and the uncertainty % to assess the potential outstanding risk.

Answer 4

Sometimes you will find non-technical project managers and to make the matters worse they work against metrics rather than common sense. However, his question is legitimate to some extent. Given a feature, you could always come up with N number of tests (after applying equivalence class partitioning, pairwise etc) and then you can group them as functional (positive, negative) and non-functional (internationalization, localization, security etc). Once you have covered all these areas then you can claim that the tests are complete. And then you can say that the execution is only 10% complete as there is a blocking issue and it is not worth continuing the test.