What goes in your definition of done (DoD) ?

Question

In agile projects, we use the definition of done to ascertain when to consider a user story to be ready for acceptance (implemented and tested). In the project's DoD we have things like following among other things:

• Unit tests implemented for new functionality and are all green (automated build and CI ensure this).
• Acceptance/story tests are written and passing.(these are in a BDD tool)
• Regression tests are green with known failures.(automated)
• Enough exploratory testing has been done to ensure the correctness of new feature and to determine it works as expected
• Unsolved defects are available in a DTS or the backlog.
• Code coverage is above x%. New implementation have not caused any regression or impact on code covergae (i.e code covergae has either preferably improved or remained static since last sprint).

As a tester do you add anything more to this or is it enough ? In case yes; what would be the other things that would help me as a tester to make a decision on when to stop testing a specific user story and pick another one.

PS: I am looking for some guiding principles other than things like testers instinct or gut feeling .

Answer 1

Ideally, the DoD for each user story should mean all tests for that user story are passing, and all automation is completed, running as part of the overall automation (as opposed to on one person's machine), and running with no errors.

Real world usually means compromises, since there's rarely enough time or resource to cover all the potential implications of any given user story.

Factors to consider with your DoD would be:

• Timeframe - do you have a 'hard' deadline or can you take the time you need? (or somewhere in between). The harder the deadline, the more your DoD will shift towards the steel thread. In that situation, testing anything that isn't steel threat becomes part of the project, and sometimes the development backlog.
• External dependencies - the more your work relies on the performance of, or information from, a third party, the more room you need to leave in the DoD to cover failure in communication or performance on the part of the external supplier. Sometimes you need to explicitly state that DoD can't be achieved without Item X from Supplier Y.
• Integration of user story does not cause regression in legacy functionality (You've added a new feature set as part of a sprint, but you haven't broken the way it used to work) - I think this is probably the only item that isn't explicitly on your list.

Answer 2

The elephant in the room: maintenance.

How do we maintain this new feature:

• what things need to be configurable?
• can we easily roll this out (what's the roll out?)?
• what support tools do we need to resolve issues with this in production quickly?

I worry that one day all of humanity will be maintining code of our forefathers... think of the children!

Answer 3

There are several useful heuristics for stopping testing. A few I can think of

• 'No more time' - i.e. we stop due to a business imposed deadline.
• Have you come to understand what you set out to understand about the feature (you do set specific goals for test sessions right?)
• Mission critical bug unearthed at which point documenting and fixing the bug becomes higher priority than testing the feature.

Michael Bolton wrote a good blog post listing these (and a few extras) far more eloquently than I can.

Answer 4

My definition of done is generally "we don't have to worry about (e.g. touch, test, do ANYTHING with) that feature again in order to release the code.

One of the better discussions I've seen on this is this presentation last year by Ken Schwaber at a Canadian company called Pyxis, it's up on U-tube in three parts, the first part is here

Answer 5

In a previous life, we used Kanban and tasks with time estimates per task to define the work to be done and the time left on a user story. For us, the DoD was when all tasks for a particular user story were all moved to the "Verified" column on the KanBan.

The only problem with this is if there is an inaccuracy in determining what tasks need to be done. We did spend a lot of time in pre-Sprint planning meetings hashing out all the tasks and time estimates so, in good faith, we expect that everything that needed to get done for a user story got done. The possibility exists that we did not task for a unit test that should have been done, or that a particular data configuration was not covered, but, assuming that we did all our due diligence up front, this worked as a DoD.

In reality, what usually ended up happening was hard deadlines snuck up on us and we had to sacrifice our DoD to be only high priority tasks. This meant some things like code reviews, automated regression tests, and other tasks got thrown onto the backlog never to be seen again.

Answer 6

I dont really have time to form an opinion of my own, but here's one from Michael Bolton, whose opinion I value & trust.

It may help you to make up your mind...

Answer 7

First off your question is extremely subjective and is completely dependent on your context so there is not a answer that will work for every scenario. That said there are a lot of things you should be considering and evaluating for relevance in your context when attempting to decide if you have tested a user story 'enough'.

1. Which test dimensions are relevant or pose a risk to your product and this particular user story: Performance, reliability, localization, supportability, testability, security, documentation, maintainability and compatibility (everything apart from functionality, this list is not exhaustive). Some of these may be contained in your BDD tests but its worth going through these types of lists to make sure you haven't forgotten to look at something important. You should know which areas are of greater interest or pose a higher risk to you product and that those areas are covered sufficiently.
2. Focus and Defocus with a measurement of that rate at which problems found. Generally focusing on one area of testing will find bugs quickly and then the rate will decrease, at that point it is worth de-focusing and looking at a different type of testing/area. Going through a series of focusing and defocusing can help you arrive at a place that is visibly defensible as a good stopping point. This is a concept from James Bach's RST. You say 'Enough exploratory testing had been done', how do you know that enough has been done? Using the Focus and Defocus Heuristic you can start to gauge this. Of course the answer is you don't know, but you can show how you made an informed decision to stop.

Other concepts that are worth thinking about:

1. Only testing user stories and not looking at testing an application as a whole is a mistake you can easily fall into using Agile. Is the project done once all the user stories are done for you DoD? The answer may be yes but it's worth thinking about.
2. Ignoring the financial implications of testing and risk when attempting to get something done. Remember a company is (generally) trying to make money from a piece of software, there is a balance in testing to maintain with this in mind.
3. Analysis all you user stories in terms of the risk each poses. It could be that in your particular context with a limited amount of test resource (in time and skills) that some user stories aren't worth testing at all while others need a far greater % of your resources.

Things on your list that aren't helpful:

1. Code coverage % targets. These aren't helpful as you can have 100% with no asserts and not be testing anything. Target achieved but no actual testing being done. Coverage is useful when developing tests to see what isn't covered and question whether uncovered areas should be testing, of course risk, time and your context will drive whether you add tests to uncovered areas. It does not mean that everything covered is 100% testing and working perfectly in all situations. Don't use it as a warm blanket to say your product/user story is done.