While writing automated test I come across situations when I include multiple verification points in one method but then I have also read/heard that one tests should make just one verification. Which approach do you follow and why?
n.b. I have my pointers to make multiple verification in one method and would detail them once I get a satisfactory answer. Looks like I am posing one more subjective/argumentative question.
I incorporate multiple verification's into one test, but only permit them at one point in the execution of the test. The test harness recovers gracefully from a failed verification (verification's should never change state IMO; if gathering data changes state, gather the data as part of the test or make that verification another test) and continues on to perform additional verifications.
Why have multiple verifications?
Why have them only at one point in the test?
Edited to add: I just realized that the infamous "Given-When-Then" format follows the same rules. You have one test (the "When" section), and after "when" is completed, you can verify multiple "then" statements using the "and" keyword. So there's one method's solution to this question.
I personally like to have atomic tests, that are designed, generally, to test only one or two things, with most verification being called explicitly called in the test case itself. That way you can easily see at the top level what is being verified, for example (from parkcalc).
[TestMethod]
public void InvalidTextAsExitDate()
{
Parking.Calculate(ParkingType.LongTermSurfaceParking, "12:00", "AM", "today", "12:00", "AM", "INVALID");
Verify.SpecificErrorIsShown(ErrorMessages.EnterCorrectDate);
}
[TestMethod]
public void OnlySpecifyEntryDate()
{
Parking.Calculate(ParkingType.LongTermSurfaceParking, "12:00", "AM", "today", "12:00", "AM", "");
Verify.VerifyResult(ParkingType.LongTermSurfaceParking, "12:00", "AM", "today", "12:00", "AM", "");
}
Whenever I have pushed verification down to the lower levels, I always keep wondering, "What is this really verifying.", e.g.
[TestMethod]
public void OnlySpecifyExitDate()
{
Parking.CalculateAndVerify(ParkingType.LongTermSurfaceParking, "12:00", "AM", "", "12:00", "AM", "today");
}
then the next level down is
public static void CalculateAndVerify(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
{
Calculate(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
Verification.Verify.VerifyResult(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
}
Full working example solution of my framework is available at https://github.com/teknologika/stax
Another draw back to combining too many things in one test, is a failure could prevent other verifications from happening.
Say you combine three verifications & the second fails. Will your automation stop? Or continue to three? And how important is three? If you can live without that being verified then combining will save time. However, if three being verified is a top priority I would put it by itself and take the time hit.
For unit tests, I greatly prefer to test one thing per test. The reason is this: When the test fails, I want the test to point me as quickly as possible to the broken code.
If a test tests multiple things, then the name of the test is not enough to point to the broken code. I have to seek additional information, such as a stack trace, which guides me to a line of test code. Then I have to look at the line of test code to see what it was doing, and in what context. Now, that's not the most horrible problem I've ever had, but...
If a test tests one thing, then the name of the failing test is often enough to point me in the right direction. Not always, but often.
For different reasons, I also prefer to make each acceptance test test one thing. But here, my reason is mostly clarity. I want each acceptance tests to describe with crystal clarity the feature or responsibility it's testing. And I want to see why this test matters. The more stuff a test does, the harder it is to understand the point of the test.
I'm more likely to relax my preference for acceptance tests, especially if the test tests through the GUI or browser. The startup and teardown sometimes take a long time. If I do more stuff in each test, then I'm executing fewer of those long-running setups and teardowns, so the suite takes less time to run. On the other hand, each test takes longer to run, and it's not possible to test just the "create a new frobozz" stuff if that test is embedded in a bigger "create a frobozz, edit it, save it, delete it" test.
I know this is going to seem a bit odd, but it depends. A lot of the testing I deal with involves completing a defined transaction(Go to this location, select quantity X of item Y, pay with form of payment Z) - for the automation I work with, this constitutes a single test.
One of the ways the team where I work evolved to gain a better idea of where problems originate is in how we structure our scripted error handling - every method in our script code has exception handling, and any exception is logged with the file name and routine name, so what gets logged might look like this: "TransactionLib.RunTransactions: TransactionLib.TransactionStep: PaymentLib.MakePayment: PaymentLib.CashPayment: Authorization form displayed when no authorization required."
Where possible, we try to recover from these so the test run can complete although it will throw baseline data out of synch and cause a lot of comparison differences as a result - the reasoning here is that if the cause of the problem is something that can be configured or is part of the application, it shouldn't stop the test run, where if an access violation or similar level error occurs, that's a drop everything and get it fixed immediately.
Our framework is designed to perform its setup, run the defined set of transaction tests (which we store in what's effectively a relational database of csv files), then do detailed baseline checks and perform cleanup. Whether this counts as running each test individually or stringing them all together would depend on precisely how "test" is defined.
Each test could use several of a large number of code routines, and we try to keep those to a single action where possible. Each test step can also involve multiple code routines, depending on precisely what's being done - again, it depends a lot on your application under test. A small web application won't need anywhere near the level of complexity I've just described: I spend most of my time working against a massively complex application with over a million lines of code.
The short version: if you can quickly diagnose problems and work out how to get them fixed, it doesn't matter how your tests are structured - although it helps if someone else can learn to diagnose and fix problems quickly too.
I incorporate several verification points in a single test method and the reasons are as below:
Before performing any action on an element I would first like to verify if the element is present on the page in the first place. There are some possibilities that the UI control is renamed or removed. The same logic applies for pages as well. I verify at every step if I am on the right page before performing any testing by verifying the title of the page. I thus have several verification points in a single test method. I first verify if I am on the right page, then whether the required UI control is present and after that I perform the intended verification point of the test case.
Certain verification points can be logically grouped in a single test method. For example, when I test "post comment feature" on facebook I have got two verification points. One is to verify if the comment is posted successfully and another is to verify if my friend can read the comment that I just posted. It makes sense to group these two verification points in a single test method since they are logically related. It is incorrect to claim that post comment works if others can't read my comment.
I assume the OP intends for "verification point" to mean a condition whose evaluation determines whether a test fails, e.g. what test frameworks often call an "assertion".
When deciding whether the code under test is ready, all you may care about is a binary answer to the question, "Does every test pass?". For all other times, your tests serve two purposes, which I will call verification and diagnosis. Verification is checking whether the software meets its requirements. Diagnosis is narrowing down the specific conditions that cause a failure.
This duality is often a source of tension between testers and developers. If a tester does nothing else, they must decide whether a product passes tests. If a test fails, the developer, if nothing else, must change some code. In between those two events, someone must determine which of the many circumstances leading up to the failure were necessary and sufficient. Armed with that information, the developer will think about how those circumstances map to code. And before that, someone may need to think about whether those circumstances happen often enough or are important enough to justify making changes.
Browse the bug tracking system of any product and you will likely see evidence of the tension between verification and diagnosis. A developer bounces a bug back to the tester because the symptoms are "too vague" or "not specific enough". A tester, having spent a great deal of time reproducing a problem, wonders why a lazy developer can't diagnose a problem for themselves.
So how does this apply to automated tests? Using multiple verification points in a single test increases its diagnostic value. If I need for f() to return true, g() to return 1, and h() to return 3 in order for a test to pass, the developer will appreciate knowing which of those three conditions failed.
There are other reasons for using multiple verification points. Sometimes a test will include "sanity checks": conditions that might point to bad assumptions in your test (e.g. availability of a resource) rather than problems in the code under test. You may also use multiple verification points because the setup expense justifies performing multiple tests at the same time, although doing so may reduce a test's diagnostic value. (In reality, we make this compromise every day; most of us do not rebuild our entire test environment for every test.)
Another reason for incorporating several conditions into one test case is simply shortening runtime. In our system setup and teardown of each test can take more than a minute or two each, when multiplied by 100's of test cases this adds up to a significant number. When I say "setup and teardown" I refer to test framework time itself so it can't be skipped by doing it once for a bunch of tests (distributed test framework over slow networks and slow servers).
Personally (and I'm fairly convinced that this the right approach) I'd always hit only one test condition with a Unit Test. Always without exception. For integration and acceptance tests I aim to hit as few test conditions as necessary to exercise the component or functionality under test. This generally doesn't mean that the checks have only one test condition and its possible that the right number of test conditions could actually be many (especially for a large test) but increasing numbers of test conditions in higher level tests can cause automated tests to become brittle and crucially, can lead to failures being more difficult to interpret. To focus my mind when writing these higher level tests, I keep in my mind the question 'what am I actually trying to check?'. If there are test conditions that are important enough to add to the automated checking but not directly relevant to what am I currently writing Acceptance tests, then I use these as the basis for new, separate test cases.
I've changed my opinion on the "rule" of only 1 assertion per test over time.
A few years ago, when entering the QA field, I considered it a great approach and almost too obvious to explain and justify. As other poster have written, it's the way to be specific about what is being tested, for failures to be more specific, etc, etc.
However I've seen that in real world situations, most significantly as test suites grow, treating this as a fixed rule becomes problematic due to the following reasons:
test suites run length. If all tests have to perform setup and teardown (they should) and only test 1 assertion this can lead to very lengthy test suites times. This is a huge problem as once test suites get slow a whole host of bad things happen (not run, slow development, increasing use of shared seed data, etc).
test suite size. If 3 tests are written that have a total of 19 assertions, changes to those tests are relatively easy. However if the 19 assertions are used to break down the tests into 19 tests with 1 assertion per test, maintaining and updating the code becomes error-prone, time-consuming and boring for many programmers.
Not DRY. When a lot of examples repeat the same setup and teardown, you start to get a lot of cut-and-paste programming and code in multiple places that needs to be updated instead of code in one place. I have frequently seen "some" tests have their setup changed without the changes being applied to other tests leading to a mish-mash of styles
programmers have different opinions and experience. I work with a lot of programmers who don't follow the "1 assertion per test" rule. Making every follow this rule would be a challenge (worth it in some organizations but not in others).
Note: An example of "multiple assertions" is that for a given UI operation one might expect
So this is 5 assertions for the same action.
One thing I do agree with and find hellps is @ethal evans mention of
"only permit them at one point in the execution of the test."
I also expect to put them at the end of the test (excepting any required teardown).
