What is the purpose of expected results for individual test case steps?

Question

I don't mean for this to be a subjective discussion and I'm trying to not come across with a value judgment. I'm genuinely interested in understanding what I see in some testing tools.

With Testrail and Microsoft Test Manager, I have seen test cases that have expected results for each step. However, I'm quite accustomed to expected results being scoped to the entire test case, not the individual steps too.

So the hopefully answerable question I have is: As a QA, how I can I leverage the concept of expected results per step?

Answer 1

There is multiple different reasons for this. I will try and explain a few.

• If the tester is unfamiliar with the system it ensures they are performing the steps correctly

By giving the tester expected results at each step it allows the tester to easily be moved from one application to another. This makes cross training a lot easier. It also helps with allowing the tester to know that the results they see are the results that they were expecting to see at that step. For example, if you are testing an API that you are unfamiliar with it would be difficult to determine if the response returned from a set up step was the response expected.

• End to End verification of all functions

By having the expected results in each individual step it assists with end to end verification of all functions within the system and makes it easier to determine if all of the system's class and functions were hit when testing. This helps to ensure that testing was thoroughly performed especially during the planning and test writing phases.

• Data Collection

By having expected results and often times, actual results, it helps to remind the tester that they may need to input data to verify that what was returned is being put into the test tracking software. This assists when it comes to defects released into production that had already been thoroughly tested during the testing cycle and somehow popped out in production.

• Defect Resolution

Often times these test cases are linked to defects. The defects can be viewed by Product owners who don't know the steps they need to perform well enough to duplicate the defect or even to developers who might work on an external system that is causing the defect in yours. By giving the verification steps and letting people see exactly what they should expect and what the tester saw you are laying the path that is needed to expedite defect resolutions and ultimately ensuring a better quality system overall.

• Avoids hit by a bus

This is hinted to in the first benefit but by having this it greatly assists in giving the details needed to be able to thoroughly test it if something was to happen, like a tester being hit by a bus. You and your bosses never know what could happen and by putting extra detail into your planning you, and they, are protected from a worst case scenario as much as possible.

TL;DR;

It's a great practice for several reasons, it's a bad practice because it takes a few extra minutes. Trust me, spend the extra time during planning and save everyone's time in the long run.

Answer 2

• A functionality requires validation of system behavior in two (or more) different states. For instance. Let's say in your SUT a user is authorized to use a certain feature only if the user has permission X. To make sure this really depends only on having this permission, you should verify at least two cases, that is your test case have have multiple steps and checks

  1. Create a user without permission X
  2. Login as the user
  3. Try to access the feature. Expect the user receives an error message "Permission denied".
  4. Grant the user with permission X
  5. Try to access the feature again. Expect the user can use it.

  It does not always make sense to split those checks to separate test cases. For instance, in this scenario you may want to verify what happens if permission is granted or revoked when the user is already logged in.

• A test with entry conditions to validation. For instance, in the above scenario you may want to check in the first step, that information about user permissions are defined correctly in DB after creation. Sure, it may blur the purpose of a the test a bit, but on the other side, it makes no sense to execute next step, if this information is missing in DB.

• An elements with multiple aspects to validate. It might be easier to read your test case, if multiple aspects of one thing are checked in separate steps. For instance, imagine you're verifying a report generated by your app. Each step then could be about verifying a value for a different column/row/cell of the report.

Answer 3

To add a dissenting voice - they are there for historical reasons as that's how people thought tests should be written and it also provided a way for tests to be 'measured'.

I view such detailed test cases as a waste of time and effort ( the answer from Paul that it's a great practice can be argued against) so I don't see what value you can 'leverage' from them

Answer 4

To elaborate on Phil's answer, the metric for test cases are always based on "pass/fail". One of my biggest complaints against Quality Center many years ago was that multiple requirements would be associated with a single overall script. That script may fail one line, but the failure is applied to all associated requirements blindly. Segmenting results to specific steps is one attempt at addressing this, but it still doesn't address the fact that a step could fail for a reason completely unrelated to the associated requirement (or pass even though the requirement is not fully met).

The reason for not documenting test cases to that extent is because any one test case can be linked to multiple requirements to varying extents or be impacted by multiple oracles when submitting defects. Whether a specific behavior is a defect would depend on the context of that particular test run. For example, repeating a test case involving adding a new user should have a different expected result if the same user is added twice.