1

Let's say you are testing a calculation done by function X.

The test case looks like this:

  1. Open the menu and select F X
  2. Confirmation dialog appears
  3. Function screen appears
  4. Enter 10 and hit ENTER - 1000 appears.

Now the thing is - what if the confirmation dialog does not show up but I can go on and the function works as expected? Because the primary objective of the test case, calculation verification, works. Making it fail because of a supporting step does not make too much sense - it would mask that the primary feature works (setting the whole TC as Failed). How should I approach this?

Improve this question
2
  • What is the answer of business analyst, on why do we need confirmation dialog in the first place??
    – Vishal Aggarwal
    Oct 3, 2019 at 1:51
  • 1
    If it doesn't serve any purpose to business, then not only we can remove it as test step , but also from the application.
    – Vishal Aggarwal
    Oct 3, 2019 at 1:53

10 Answers 10

Reset to default
2

Honestly, in this example, I'm confused about why the confirmation dialog should appear at all. Isn't it just a useless step that slows down the user? There might be some reason for that that I can't get from the short description, but I'm a bit suspicious, that's why I'm bringing it up.

And I think that's exactly what you should do with the rest of your team. As a Tester, you should point out all these areas where you feel there's something fishy. In this case, you're thinking whether the confirmation dialog is important for the test, I think you should take it even further and ask whether or not the confirmation dialog is important at all.

Improve this answer
2
  • Puzzle - that was my thinking too. If the step is something you are willing to over look in testing even though it technically 'failed' what is the purpose of the dialog? Or is it just an extra UI / click users have to do with no real purpose. In which case advocating for its removal might be the right answer.
    – CKlein
    Oct 1, 2019 at 11:12
  • Well, in reality this can happen for several reasons - e.g. the confirmation has to be there (regulations, mandatory behaviour, etc.) so you cannot remove it but because this test is focused on the main feature, whether or not the dialog step fails does not matter.
    – John V
    Oct 3, 2019 at 13:10
1

I guess this might depend.. First of all everything has its own cost. If you have lots of such tests it might take lot of time to raise each such defect. However the defect might get low severity and low priority and will never be handled.

Another thing is that the use case itself might be very unlikely, or the area itself is to be reworked as per team plans so the dialog might be changed soon and those changes are being delivered by the moment (what could probably cause the issue you observe).

Otherwise I would tend to consider tests failed even if a single step (whatever it be) fails.

Improve this answer
0

Actually, you have multiple tests on the above description.

1.1 - Open the menu

  • Here you have hidden all complexity of start-up the application

1.2 - and select F X

  • Here you test that an even is trigged when a button is selected

2 - Confirmation dialog appears

  • Here you test that the dialog appears when an even is trigged.

3 - Function screen appears

  • Here you test that the screen appears when an even is trigged.

4 - Enter 10 and hit ENTER - 1000 appears.

  • Here you test the UI for the for dealing with input and output displaying and the backend calculation of certain input.

For optimize feedback, a test should fail for only one reason. The test for (2) should be only concerned with triggering the event and investigating the reaction, and fail only if this reaction is not as expected. Other tests should fail when the reason them to be fail.

Improve this answer
4
  • Well, but in practice, imagine you want to run only TC no 4. You need to get there somehow, especially if you are a new tester who does not know the system. So sometimes the steps are there just to guide/describe what you need to do.
    – John V
    Sep 30, 2019 at 18:20
  • For the UI part, you can simply instantiate the UI, mocking out the backend. For the backend, you simply instantiate the backend component and interact with it.
    – João Farias
    Oct 1, 2019 at 5:13
  • But this is about manual testing.
    – John V
    Oct 1, 2019 at 5:55
  • 1 - The question was wordy in a way that seems focused on algorithm checks: Predictable behavior and output assertions. 2 - Nonetheless, the strategy in the answer is valid for exploration: If you want to investigate the risks on the calculation input UI (step 4), you can write a script and boots only a specific UI component and from there you explore. The point made in the answer is that the exploration area of the four steps is, unnecessarily, large: Different risks can be investigated in isolation.
    – João Farias
    Oct 1, 2019 at 8:18
0

What you should do depends on the processed followed by your team and what kind of testcase we are dealing with.

What should happen in any case is that a defect is raised, because either the software contains a bug (if the confirmation screen should have been shown), or the testcase is incorrect in expecting the screen is shown unconditionally.

If your processes say that the testcase must be failed before you can raise a defect, then you should fail the testcase.
If it is a manual test and you can freely raise defects, then it is your call if you want to mark the testcase as failed or not.
If it is an automated test, then most frameworks consider all steps as critical, unless you explicitly code a step as having optional behaviour. That should only be done if it is explicitly confirmed in the requirements that the behaviour is really optional.

Improve this answer
0

If step is not important would not have been included in the test.

Failure of any step, should result in the entire test failure.

Having or not having workarounds or how the failure impacts user experience might only define the defect severity.

But

The defect has to be registered.

Improve this answer
0

Defiantly this should not be ignored, but the approach should depend on how "up to date" the test case and/or the documentation is. The course of actions in my opinion should be:

  1. If you have no doubts about the test case and the business requirements - fail the test case,

otherwise:

  1. Ask your project team / test manager - maybe this step is obsolete as requirements did change and the test case should be updated (asking people around is always the fastest way to obtain feedback).
  2. If there is nobody to ask - raise a minor bug, so it is not forgotten, ask in comment if this is still a valid requirement, or should the test case be updated.
  3. If your team does defect calls - raise a minor and ask if it is a valid bug on defect call.

Unless you ask someone and solve it immediately, the test case should remain failed, if you do not solve this now, it will appear again on next iteration of test case execution.

Improve this answer
0

Because you have 2 test cases; that's the only reason this looks confusing at all.

  1. Functional test case: Do I get a correct (correct may mean correct-enough, example 999.9 might be acceptable for your domain) answer to my question.
  2. UI experience test case: Can user see the menu, click it and get to a entry screen. Does the user get confirmation dialogs and help instructions along the way and end with a nice big arrow (the arrow must be pink BTW) pointing to the answer.

The moment you split these, the business value get separated nicely from the accuracy problem too. (BTW Some answers are identical to mine, but I have 0 reputation on this /stack, so unable to comment/give input.)

Improve this answer
0

This is a common case in a software testing company.

In this scenario there can be 2 different point of views.

  1. You can review the important step again and check if it is a mandatory step else, can refactor the test case as the desired result is already achieved.

  2. You should fail the test case as an important step did not execute correctly even though desired result is achieved. Reason for this would be to take in consideration that the referred test case would fail due to the multiple other steps or while performing regressing testing for some other defect and this defect could only be fixed due to the changing the referred important step.

Improve this answer
0

I will prefer to put the TC status BLOCKED, open a bug for the step that fails and when the bug is solved put the whole TC status PASS.

Improve this answer
-4

In my tests, I report after each step. I'm using TestProject, so they have a great mechanism for the reports, but you can definitely do it by yourself (for example using logs mechanism). That way, you don't have to fail the entire test if you think that this step is not critical, but still, document the failure.

Disclaimer: I am employed by TestProject.

Improve this answer
7
  • How does this answer the question?
    – Niels van Reijmersdal
    Oct 3, 2019 at 8:47
  • Did you read my answer? I said that he needs to report the step failure even if it's not important. I also suggested a way to do it.
    – Amit Yahav
    Oct 3, 2019 at 11:10
  • Maybe, but you barely explained your reasoning. I guess allready stopped reading after the self promotion product placement and the word reports.
    – Niels van Reijmersdal
    Oct 3, 2019 at 11:49
  • You can call it self-promotion, but I answered him and provided a valid solution.
    – Amit Yahav
    Oct 3, 2019 at 11:55
  • 1
    Hi Amit. To help prevent you getting more spam flags against your name, I have edited in a very basic disclaimer. Please be sure to disclose you affiliation when you reference your employer's product in an answer. You will find more information in the Help Center (sqa.stackexchange.com/help/promotion)
    – Kate Paulk ♦
    Oct 8, 2019 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.