3 I guess this example was more backing up the OP's intuition than "inspired" by the book section
source | link

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's potentially a spaghetti, unmaintainable codebase if it requires significant testing of unrelated features even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The example I gave above was inspired by the part on "Instant reviews" in Chapter 16 is very related to the example you gave that I alluded to above, about observations that tell you much more about the quality of the product than the actual test (or review) in question.

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's potentially a spaghetti, unmaintainable codebase if it requires significant testing of unrelated features even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The example I gave above was inspired by the part on "Instant reviews" in Chapter 16.

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's potentially a spaghetti, unmaintainable codebase if it requires significant testing of unrelated features even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The part on "Instant reviews" in Chapter 16 is very related to the example you gave that I alluded to above, about observations that tell you much more about the quality of the product than the actual test (or review) in question.

2 Qualify point in example
source | link

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a complete redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's potentially a spaghetti, unmaintainable codebase thatif it requires significant testing of unrelated features even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The example I gave above was inspired by the part on "Instant reviews" in Chapter 16.

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a complete redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's a spaghetti, unmaintainable codebase that requires significant testing even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The example I gave above was inspired by the part on "Instant reviews" in Chapter 16.

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's potentially a spaghetti, unmaintainable codebase if it requires significant testing of unrelated features even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The example I gave above was inspired by the part on "Instant reviews" in Chapter 16.

1
source | link

I think the general answer is "be reasonable". Just because you're doing blackbox testing doesn't mean you should do extra useless testing--in this type of situation, you're likely better off looking at how the code is implemented and deciding on the scope of testing accordingly. I've seen too many blackbox tests that were long and painful to execute but didn't add any value because they were doing lots of checks that each "looked" different, but in reality were testing the exact same code path over and over.

Now, if you looked at the code and found that there really is different code for displaying the exact same message based on which type of user is logged in, then you would need to back off, question your assumptions, and do a lot more testing. But probably in that case a complete redesign is probably a much more valuable use of time. In that case, a simple code check would have revealed a far bigger problem with the product than whether this text always appears correctly, namely, that it's a spaghetti, unmaintainable codebase that requires significant testing even for such a small change as the text of a popup.

If someone is pushing testing all the possible cases for this change because "we should test everything", have them read Perfect Software; and other illusions about testing by Jerry Weinberg, which is a very eye-opening book on how to approach (and not approach) testing. In particular, Chapter 3 "Why not just test everything?" makes some very good points about why "testing everything" is a common fallacy. The example I gave above was inspired by the part on "Instant reviews" in Chapter 16.