Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I have a data migration project going on. There's a semantic mapping tool in the middle to translate between two DBs. I'm testing that.

However, some issues that I find are fairly...blocking. E.g, one that causes the wrong number of values to appear in every column in the new database. Fairly big.

As a result, any other tests I write that get masked by this issue I can't know if the test is right yet, or if once the blocking issue is fixed we may see more issues, or whether everything will pass.

The project manager wants to know progress and when I said it's tricky to tell because of this massively blocking issue, says "put it this way, how many tests have you written?".

To me this is a horribly inaccurate measure, as the tests could be wrong, or new bugs could cause new tests to need to be written, or we may run into even further blockers, let alone the fact that the tests will need to be re-run.

He finally insisted that I tell him how many tests I've written, at which I could say "80%, but..." and he said "well there's a metric!". I tried pointing out why it's not a realistic one to measure progress though.

Are there more legitimate ways to report progress to project managers when you have big blocking issues potentially masking further problems?

share|improve this question
    
You mentioned "I can't know if the test is right yet" and "new bugs could cause new tests to need to be written". To me, this points to a lack of planning on the PM's part. If you're not sure if a test is correct, get clarification from the PM or dev. If you're unsure of some of the specifics on how the migration should happen (which could call for more tests or invalidate tests), get specifics from the PM or dev. You should be able to write nearly all of your tests with no code written. If not, your project is sorely lacking in planning and that is on the PM, not you. –  Sam Woods Jul 5 '12 at 16:26
    
To clarify, it's an automated test. So it could be buggy. But it's hard to debug when there are so many bugs in the way. So there could well be a problem with the test that will be uncovered when some of the fixes are complete. The documented tests are written - I did a detailed test plan at the start and got them signed off. –  Mark Mayo Jul 5 '12 at 16:29
    
You say "I tried pointing out why it's not a realistic one to measure progress...". What response did you get? –  user246 Jul 5 '12 at 16:43
    
Not much response, he just wanted a number :/ –  Mark Mayo Jul 5 '12 at 16:55
add comment

3 Answers

You need to meet your project manager half-way. Using an unrealistic metric does not help either of you, and you are right to be concerned about that. At the same time, you need to avoid describing your problem in so much detail that your project manager will not understand it. Given what you wrote, I think I would tell the project manager something like, "If you want an accurate status that you can measure against your schedule, you need to break down my work into N parts (where N is preferably a very small number): design, code, debug, etc. Here is where I am against those N parts. And here are some risks to my continuing to make progress."

If you already did that, and you are not satisfied with their response, you should amend your question with that information.

share|improve this answer
add comment

The metrics can be divided so that there is reportable status. 1) List of test conditions (i.e., test cases). Do you believe that you have a complete list written down? If so there is one metric (identification of the test conditions). 2) From the list of test cases, how many are completely fleshed out as far as what needs to be done to run the test? Your blocking issue probably prevents documentation of adequate detail on many of the cases you've identified. This would be a second metric (development of test case steps). 3) Test execution would be a 3rd metric. What %, if any, have been executed and passed. If there are none that wouldn't need to be re-executed after the blocking issue is fixed, this part is 0% complete. If there are some tests which aren't impacted by the blocking issue, you "may" be able to count those as completed.

share|improve this answer
    
You may want to identify approximate percentages for how long each of these phases takes (helps to have historical data) and an estimation of how much time the total task would take (again, some historical data helps a lot). If you can mine any data about how long small vs. medium vs. large projects took in this past, that's a good place to start. Then you can make a prediction on how the current project stacks up against known projects. It's just a prediction, it's not a commitment. –  Jean Marchant Jul 5 '12 at 17:29
add comment

Sometimes you will find non-technical project managers and to make the matters worse they work against metrics rather than common sense. However, his question is legitimate to some extent. Given a feature, you could always come up with N number of tests (after applying equivalence class partitioning, pairwise etc) and then you can group them as functional (positive, negative) and non-functional (internationalization, localization, security etc). Once you have covered all these areas then you can claim that the tests are complete. And then you can say that the execution is only 10% complete as there is a blocking issue and it is not worth continuing the test.

share|improve this answer
1  
+1, although I would add that you may be able to continue executing some of the tests (maybe partially) even with the blocking issue to try to discover any defects as soon as possible. –  Sam Woods Jul 5 '12 at 16:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.