Should test cases contain more than one step with expected result (ISO 29119 definition)

Question

There is a big dispute in the software testing world. In general, a test case is a pair of an input and an expected output (or as many books say, it mostly has just a single step: do sth and check it). ISTQB seems to be compliant with this. Of course in order to perform a test, one often has to follow certain actions/steps (start a module, open a window, navigate somewhere...) but these supporting steps are not intended to have an expected result if the test is verifying e.g. correctness of a calculation. But most testers do the following:

TC1 - Verify that file bigger than 10 mb cannot be uploaded
Step 1 Open a window - Ex:Window is opened
Step 2 Invoke the file dialog - Ex: The dialog is shown
Step 3 Click on the Browse button - Ex: File browser window appears ...
Step N Select a file bigger than 10 mb and confirm - Ex: An error message appears

Steps 1-3 are generic and possibly repeated for any similar actions involving a file browser. They are necessary for the final step but I do not think they should be verified in the same test case. Because if step 2 fails, the test case fails even though the actual test condition checked in the last step is correct!

Strictly speaking, I would consider this a test procedure as it actually consists of a few test cases.

Or is there something I am missing? I do not have access to ISO 29119 but I have read that it defined input as "a comprehensive descriptions of the actions/and or data" - which would mean all the steps would be performed as a part of input and not verified separately.

What is the usage of steps then, does it really make sense to use them as I describe?

Answer 1

+1 for it depends

If the upload operation in your example can only be done through the mentioned failed menu than it is difficult to not fail it.

OTOH maybe you are part of the "upload team" and all you want to do is verify this small operation and not end to end integration, so failing the test is out of the question

Another approach (I use it with Microsoft Test Manager now) is having tests Passing, Failing or considered as Blocked if something is blocking you from testing the thing you wanted to test. This usually requires your test scripts to be build the right way, for example using common setup() and tardown() functions as an easy way to separate the main test logic from the surrounding environment

Answer 2

In general, a test case is a pair of an input and an expected output (or as many books say, it mostly has just a single step: do sth and check it).

A test case or a simply "a test" is an experiment we run in order to reveal some information about a product or service.

one often has to follow certain actions/steps

Only if you are taking the approach of (heavily) scripted testing. For example if you take more exploratory approach you try several different ways to test an upload file size limit. For more research look at: Exploratory Testing.

But most testers do the following

Most testers where? That you work with? Again I would imagine most people who do testing take a primarily exploratory approach. I don't have any hard evidence of this but in my experience and from those I talk to (including a lot of programmers), they tend to work in a more natural form.

Should test cases contain more than one step with expected result? What is the usage of steps then, does it really make sense to use them as I describe?

If I had to guess I'd say you are using a "test management system" that lists things out in a scripted way? The truth, is not every test you are going to run will have an expected output because you may not have any way to understand what Oracle is appropriate. (An Oracle is a way to recognize a problem.)

Answer 3

Generally speaking these concepts are mostly misunderstood and the standards, if any, are not followed by the companies in most cases. I would, at least, from my experience can tell software are so versatile to put them in such categories/boundaries. Additionally, as a Senior Q.A I have been practicing testing the scenarios/test cases out of box with having the goal to find out what happens if I do that or this. In order to replicate the possible user's behaviors there are thousand different scenarios I always come up with which of course have some expected result. In that case, I do not think about test cases or steps to be frank with you.

Additionally, when I write automation test cases I tend to keep each test(often referred as test cases) independent, which is also the best practice in this industry. And, test is followed by an Assert, which is the expected behavior(positive and negative) by the software. If otherwise happens, the test usually marked as fail. I generally consider the end result to decide if the test should be marked as pass or fail. However, if it fail somewhere in the middle, say the test steps for the sake of argument, I investigate to point out the issue if related to something unexpected and if so marked them as fail. So, in that case I can say, the software did not behave as expected and, thus it failed. So some kind of result is expected from every step. It's also worth of mentioning that multiple asserts for every steps in test is considered as bad practice and I do not follow them either. Hope this helps!

Answer 4

Different testers have different ideas about test cases. Sometimes, a test itself is called a test case

• A test case is one particular instance or variation of a test or test idea
• A test procedure is a way of performing a test.
• A test activity is a line of investigation that fulfils some part of the test strategy. It can encompass many test cases.

There is no best practice or an ideal standard because it all depends on the context. The term is used interchangeably in different organizations & sometimes in different projects within the same organisation. Regardless of what terminology you use, its important that everyone in your organization or at least your project speak the same language to avoid confusion.