Test case design standards- friend or foe?

Question

Currently we use an agile team based approach to testing, where each team is responsible for their specific area of a product. While we have found this to be effective and easily manageable, over time teams have adopted the best test design that suites their own team. Now as we have grown its become difficult to manage test coverage, GUI automation, and cross team testing across the entire product.

I'd like to implement some kind of standard without bogging down testers with test case design. I'd much rather them spend more time in the product exploratory testing than writing test cases. Is there a light weight design standard I can implement that anyone could recommend that fits well within an agile shop?

A second question is, are standards the right approach in the first place? I'm concerned about attempting to enforce this behavior across an organization.

Answer 1

Answering your second question first: yes, you absolutely need test case design standards. They don't have to be monsters. As with all things agile, they need to be enough.

For your first question there's no "right" answer, but I can give a few guidelines. Your standards should be based around a kind of test case triage: first priority is being able to set up and verify the user acceptance tests for each story. Second is anything else in the steel thread. After that you look towards the 80/20 rule and critical boundaries between areas of the system.

You'll want some guidelines/standards for what makes a good automation test case to add to your existing automation. It's fine to have a whole range of different automation approaches as long as every automator across your testing teams can pick up and add to any of the automated project at need.

Some of the standards that I use where I work (which isn't entirely agile, has an application codebase of around 2 million lines of code some of which dates back to 1996, and adds new features at a mind-boggling pace) are:

• Whatever it is, include setup instructions. You can't guarantee the next person to deal with this knows how to configure it
• However you document it, state what you're expecting to happen and what constitutes failure. The rest of the team aren't mind readers.
• Focus on verifying and validating expected functionality first. Then check anything you know is related and list it. After that hit the negative cases (with the caveat "if there's time")
• Exploratory testing gets summarized after you've done it to allow any problematic or awkward cases you encounter to be added to the knowledge base of the team.
• Cross correlate as much as you can. If you know it involves five other modules, make sure you list them all so when someone goes looking for test cases involving one of them, they find yours.
• Clear and simple beats fancy every time.
• Know your audience: where I work we assume basic knowledge of our applications because we're a business-to-business shop and we will always have trained users. If you're dealing with a public-facing website you might want to assume no knowledge. This affects the level of detail you need to put into your design and documentation.
• Have an agreed-upon repository. You can't begin to work out your cross-module and in-module coverage if there isn't a central repository where all the test cases you've got are searchable by module, feature set, and anything else you find needful.
• Global test cases probably don't need to be in the repository (things like spelling checks, tab order on forms and so forth - testing those is one of those "habitual do it every time" things - or it should be. If it's not then yes, the global cases do need to be in the repository after all)
• If the test requires special equipment, make sure everything that's needed to install and configure the equipment is in the test case (including where to get the equipment in some cases).
• If the test depends on specific data include where to get the data in the test case.
• Have a mutually agreed-on module listing. What's in the list doesn't matter nearly as much as the list being something everyone in the team can work with.

There are some good free tools, such as test link, that will allow you to build a test case repository by module and identify your core regression cases, your automatable cases, and your gaps. I can't give more than that because my workplace is in the process of introducing this. We've been working with directory structures which has proved to be impossibly cumbersome - with ugly results. The sooner you have the repository, the better, knowing what it's like without one.

This is a long way from a complete answer, but it's a starting point to work from.

I hope that helps.

Answer 2

I think you're discovering the answer to your second question yourself, but to be less cryptic yes I think test case design standards are the right way to go.

With test case design we have to remember that when they were first proposed and used the vast majority of testing was carried out using a traditional waterfall/staged approach, which essentially means that the testing was done at the end of the project. Thus the test team could take their time (so to speak) and ponder over the most efficient way to produce the minimal set of test cases that provided the maximum coverage.

However in the testing world we live today development methodologies such as Agile exist that require people to identify, construct and validate solutions almost on the fly - when compared to a staged approach to software development; And as such the tester cannot spend 2 or 3 days fine tuning the test suite into a minimal set.

Where am I going with this?

Well I still think design standards are just as applicable in the Agile methodology you just need to look for different signs and maybe use terminology that is suited to Agile. For example if you use story cards to explain the user journey then you can put a stake in the ground and state that a "stick man" will only ever represent a process carried out by a real person - thus you've created a design standard that everyone can relate to.

What I believe is that regardless of the development methodology used you should be able to define and agree upon the test case design standards that help to make people more comfortable and therefore more effective.

Hope this rather long answer helps in some way.

Steve.

Answer 3

Steve,

Great question.

There are many ways to attack the challenges you bring up. In my opinion, you're correct to want to both (a) avoid too much procedural stuff, but also (b) introduce a way to create some more communication between team members (and perhaps add consistency and thoroughness?) in your test design process. Kate has already mentioned a lot of great points. To offer an additional perspective,

From a "how much detail is appropriate?" standpoint,

One of the key things to get right is not bogging your tests down with too much unnecessary detail. I like using this 2x2 grid to explain useful considerations when determining how much detail is appropriate point:

From a "which approach should I use to manage my test cases as they're designed and documented?" standpoint,

I would advocate a fairly light-weight approach that appropriately balances clarity (so anyone looking at the documentation can QUICKLY understand what is being covered) and flexibility (so people on the team can make changes quickly). I gave a presentation last year on a variety of successful approaches that some teams have used, below (including mind maps and kan ban board-like solutions). For you, I'd recommend taking a look at Paul Holland's approach and the approach used by the team at Mozilla / via Marlena Compton. The slides from the presentation are available here:

From the dual considerations of "how can I make each of my tests cover as much as possible without repeating things that have already been tested (and make it easy for each group to see what other groups are already testing)?" and "how can I avoid the common problem of creating a system that leads to an ever-increasing set of regression tests that take ever-longer to test with each iteration?",

Disclosure: I created the Hexawise test case generating tool.

I would suggest considering using a test case generating tool like Hexawise or AllPairs that would allow you to quickly create (and/or quickly modify and re-generate) tests with the push of a button at the "model" level as opposed to developing a test design approach that requires you and your teammates to make changes at an individual test level because: (a) making changes to individual tests would take a lot more time, (b) a Hexawise-generated tests approach is more suitable for Agile settings like yours where tests will need to be regularly updated, and (c) a Hexawise type of test-generation approach could prevent your set of regression tests from expanding much because generating tests with Hexawise could allow you to test a growing list of new features by building most of the feature-related functional tests into an existing set of tests (e.g., scenarios exercised in existing tests can get more varied and hit new functionality vs. the more common approach of testing new functionality via adding an ever-larger list of tests).

Answer 4

I don't agree with your observations of waterfall vs. agile test quality, While the general conclusions are correct. Iterative test cycles (either agile or plain spiral) gives the testers a better chance to sharpen their tests, I worked in waterfall, agile and iterative projects and the first release to test is always the same- confusion, mess, and nothing that works out-of-the-box, but when you still have time until the final release you also have time to fix all those environment and test issues.