The meaning of failure - what is the flaw in our process?

Question

A company I've started working for uses a QA process I haven't come across before, and it seems flawed to me. I'm keen to know what we should do to fix this process.

We use a JIRA workflow. After our developers have sent a new/fixed software item to QA, we will fail the item if the new/fixed functionality is incorrect. We will also fail the item if regression testing fails, indicating that the change has introduced a new bug.

What I haven't come across before is our process if any of the following come to light during testing:

• A baseline bug is discovered (i.e., a bug that was already present in the software before the new version was created);

• The usability of a new feature is realized not to be ideal, so that although the new version meets the specification, we need to specify it differently and make a new change to the software;

• A new business requirement comes to light, such that we need to re-spec and make a new change to the software.

In cases like these we are encouraged by management not to create a new work item, so we regularly have to return such items to the developers as "failed" in order for the new software change to be made. This makes it somewhat complicated to keep track of changes in JIRA, as any one JIRA item can become somewhat confusing, with different issues at different stages of completion logged against the one item, whose title may no longer be meaningful in the context of the current focus. Also it does create bad feeling with the developers, who understandably feel their fix has not "failed", and can even end up being disciplined by the company for items taking longer to develop than originally estimated, or the number of times their items have been returned as failed from QA .

What change should we make to our QA process to overcome this issue?

Answer 1

We can't tell you how to fix your process because it depends on your own circumstances.

That said, this does not sound like a hard problem solve. It sounds as if your Jira configuration is not expressive enough for your workflow. If it were me, I would do start by getting agreement on which workflow problems need fixing. It sounds as if you already have an opinion about that. Some problems are probably more urgent than others, so it would help to prioritize them.

Next, you need to decide whether to address those issues in Jira. I believe you can add fields and customize the workflow in Jira. You might also decide that some things don't belong in Jira. For example, you might choose to keep the feature list in Confluence instead. Of course you haven't said whether someone is available to customize Jira. If not, you will need to solve the problem in some other way.

Answer 2

The second and third bullet points are arguably not bugs at all. They're new feature requests. Invent a list where anyone -- testers, programmers, marketing, subject matter experts -- can suggest a new feature. Sticky notes on a wall will do. That new request gets prioritized like any other request. (By contrast, a policy might be that regression bugs must be fixed at once.)

For the first bullet point, Jira has a relation called "discovered while testing". That means you have the bug in the system, and you can track when/why it was found, but you don't imply the bug is with the new feature.

Answer 3

There is always wiggle-room with these sorts of points, but I would say you are correct in thinking that the first and third issues should be treated as new issues (bug and feature request respectively). Including logically separate issues in one JIRA issue is only a good idea if they are so tightly bound that neither one can reasonably be fixed without fixing the other.

As for the second one, it depends. If it's a question of the users coming up with a bright new idea, then probably better to make it a new feature. If on the other hand (and I have met this frequently) a programmer manages to fulfil the letter of a specification while severely misunderstanding the spirit of it, then I can understand people asking to get it sorted within the scope of the same issue.

HOWEVER: I have seen this sort of "stuff it in the same issue" mentality arise as a result of excessively rigid release procedures which insist that the definitive list of issues in a release has to be fixed in advance. If that is the situation you are in, then that is the problem that needs to be fixed first.

In all cases, a spirit of co-operation between the different groups and a desire to produce a good result is worth much more than the precise details of the formal procedures.

Answer 4

Basic flaw in your process is that your process creates adversarial relationship between developers and testers:

it does create bad feeling with the developers, who understandably feel their fix has not "failed", and can even end up being disciplined by the company for items taking longer to develop than originally estimated, or the number of times their items have been returned as failed from QA .

This in IMHO fatal flaw. Punishing developers for something they cannot control like requirement change will make the best developers leave. Maybe your management thinks your developers are not competent enough to try the best without punishment - and maybe they are right, as a result of such treatment, competent developers should leave.

Company should hire competent developers, and assume that any bug which sneaked to production was mistake, not malice. Have post-mortem about how such mistake avoid in the future, not who is to blame and how to punish him or her.

And yes, new features and improved understanding is new request, not a failed bug (especially if developer would be punished for the "failure").

Goal both developers, QA testers, and management should be to deploy best product to satisfy most important user's needs, in available time and budget.

Answer 5

If a new feature doesn't pass a 'usability' check I'd expect to negotiate how to log the incident with the developers. If it just needs a minor change (our specs are not intended to tell the devs exactly what to do but to allow them to make the best interpretation of the requirement) then I'd expect it to be added into the existing work item. If it's a significant change that has to go back to Analysis I'd expect a new work item. If the item has caused a related area of software to fail (we have a large degree of inherited code in some of our legacy systems) then I'd expect this to go against the original work item.

In the main, I base my decision on "will the system work without this change" - if it works it's a new item, if it doesn't it's the original item.