Style of acceptance test cases? (BDD)

Question

I've been struggling a bit when writing test cases. Our team follows a 2 week sprint, and I'm writing acceptance test cases on these user stories.

I've tried the BDD format (Given/When/Then) and it feels...short. I'm not automating everything (although what I do automate I automate after development is stable, and I use cucumber). I'm not sure if this is the right way to go (Although I love the BDD methodology and I want to try it).

Then again this is also an "outline style", a kind of a "Verify X works this way" with a sort of checklist.

I'm not really sure which way makes the most sense, and really what the advantage/disadvantage is. I'm not really looking for a "Which is better" but more of a "Which makes more sense for BDD" or "advantages/disadvantages"

Answer 1

I would recommend doing both approaches, for different purposes.

Given / When / Then is good for testing communication. This format brings assurance that what you are testing is what the business owner / requirements writer intended. The bugs mainly caught by these tests, IME, are ones where the business owner intended X and the dev did Y. Often, these tests (if the devs read them) will "catch bugs" before they are implemented, because the dev will come to tell you that the test is wrong. A conversation with the business owner usually clears up where the issue is, and often it is a communication issue. These tests will also catch basic "the feature doesn't work" issues that somehow leak through the developer's unit tests.

However, Given / When / Then tests often carry a substantial amount of overhead. First, writing the G/W/T format tends to take more time than a checklist, especially to write this format well. Second, to get the most value from G/W/T tests, you need to ensure that both the business owner and the developer read them before coding starts. Often developers, especially, horde their precious work time - the less you have for them to review, the more likely they are to be involved. Finally, IME maintenance for tests implemented with G/W/T languages like Cucumber tends to be more costly than e.g., JUnit tests, mostly because there is the additional "English" layer on top of the libraries. Moving from English to code effectively has some quirks that do create additional maintenance work. It's easy to end up with test steps that seem to do one thing in English but actually do another in code, especially if you have multiple test implementors. In general, you want a test pyramid - acceptance tests should be the smallest group of tests, then system integration tests, then component integration tests, then unit tests.

Personally, I use the Given / When / Then format for only those things that have clear requirements around them that the business owner cares about. For some code, this will include mainly happy path scenarios while the engineers will be the best ones to determine how error cases should be handled. In other cases, there are requirements around error cases that the business cares about deeply and additional Given / When / Then style outlines will be needed for those cases.

I usually then use a list or checklist to cover the other scenarios or test work that the business owner doesn't care about. This is mostly error cases, but will also include things like logging, metrics, documentation, very technical aspects of the feature like queue processing performance, and so on.

Answer 2

The more detailed the test case, the more time you will spend on revisions as the software evolves.

The less detailed the test case, the more you are leaving it up to your testers to read between the lines.

Maybe you should try it both ways and see how it goes.