Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

The company I work for conducts quarterly performance reviews. As a member of our small QA team I've been ask to help define this process. A lot of what I do is identifying issues and collaborating with the devs to reach a solution.

What are the deliverables of the QA department? Would any hard metric be useful?

I fear that if a metric were ‘number of issues identified’ (for example), that might lead to a focus on creating a large number of issues with little value.

Any suggestions would be appreciated.

share|improve this question
    
Possible duplicate: sqa.stackexchange.com/questions/3381/… –  user867 Jan 14 at 2:35
    
Why do you think that metrics for bug reports is the only thing that QA department delivers? What about such deliverables as CI, tools for troubleshooting issues, etc.? –  dzieciou Jan 15 at 8:03
    
@dzieciou That idea was not my intention. I can modify the question if it puts off that impression. –  David Walz Jan 15 at 19:25
    
@DavidWalz, no, your question does not puts off that impression, but most answers are. Looks like to most people feedback is the only thing QA department provides. –  dzieciou Jan 15 at 20:17
    
As a member of the team, you're tasked with defining the deliverables of the team? To me, that seems the responsibility of the manager. That isn't to take away from the validity of the question, but these are the kinds of things that should have been laid out before the team was even created. –  corsiKa Jan 19 at 16:26
add comment

8 Answers 8

The deliverable is information about the system.

The purpose is to help people make better decisions, based on information about the system.

share|improve this answer
    
This is a good answer that clears up a major misunderstanding about the field of QA, and I'd love to see it expanded. There are many common types of information delivered to help make better decisions, and a brief analysis of the circumstances in which to deliver each would be wonderful (especially as this skill of knowing what to deliver when is key to developing beyond a junior or mid-level QA engineer). E.g.: bugs, risk analysis, advice on quality practices tailored to that company or team, metrics, and tools to gain quality information automatically. –  Ethel Evans Jan 18 at 0:04
add comment

The problem with any form of hard metric in a field like software QA is that - to use the answer I give so often here - it depends. Each hard data point is really only helpful in the context of the development project it links to, and even then there are variations.

I'd suggest you take a look at the answers to the question user867 suggested as a possible duplicate, including my answer.

Deliverables should, in my view, consist of information about the system, as Dale Emery said. If you have use cases or user stories, some things you would consider for each project are:

  1. link use cases/user stories to test cases covering them and report their status
  2. list which test cases are automated and why those cases were automated (in general the scenarios that are most likely to get the heaviest use are the best candidates for automation)
  3. the state of overall product regression if you have it (this is particularly important when you're dealing with development against an existing product)
  4. areas that weren't tested, the reasons these were not tested, and the potential risks exposed. It's very rare that everything can be tested, so there's always going to be a need for testing triage.

Where there are no use cases/user stories (which is far from ideal, but certainly happens), you'd want to look for these items:

  • a breakdown of the product or project by functional area is essential because without use cases/user stories the specification is by default the working software.
  • test cases linked to the functional areas of the product/project.
  • items 2, 3, and 4 from the use case list above.

If you can data-mine the issue tracking database, some areas to consider are:

  • Issues found by testers and issues found by customers. This should not be a straight ratio metric: instead look for clusters of functionality in customer issues and whether your testers have noted that area as a problem region but have not (usually due to time constraints) been able to build in regression around that area.
  • Issues reported by testers and not corrected, then later reported by customers - here you want to look at why the issue wasn't corrected (in my experience it's usually a case of "nobody would do that" - and inevitably someone does).
  • How many of the customer-reported issues are core problems and how many are edge cases. If your test team is doing well, you should find that most of the customer-reported issues are edge cases with respect to your entire customer base or to the actions that are problematic (this doesn't make the issue less problematic to the customer - the goal here is to look at whether you would reasonably expect your testers to have found the problem before release. With a good test team and a reasonably mature automation effort, the answer will usually be "no").

Some other things to consider are the nature of the information the testers generate: do they routinely provide information on complex configuration that later becomes the core of the documentation team's work? Do they provide a list of "gotchas" or caveats about the software, such as whether a particular feature should be used in conjunction with other features, or required settings? Do they warn of fragile software that works only if everything is exactly right? Do they try to ensure that the software guides the users to the correct/preferred actions?

It's worth remembering that software development is an odd mix of science and art, and testing is possibly even more so. The science part can be quantified, but the art aspect can't, and like art every software development project is different.

share|improve this answer
add comment

from my own experience, the deliverables can be vary from one project to another. For instance, I was involved in a very new feature that has to ship within couple of months. Due to time pressure, I wrote the deployment guide document for our sales engineers so that they will know to how to deploy in customer's environment without banging their heads against the wall. That has become the key deliverable for the project from QA.

In short, apart of the usual tester's routines like raising quality bug report (rejection rate is low), look for gap/areas that QA department can bridge e.g., clarity of product requirements, documentation and etc.

share|improve this answer
add comment

I can think of a number of possible deliverables. Pick those that your teams and stakeholder need.

I would say QA deparments provides

  • Information/feedback about the product, e.g,

    • bug reports
    • test reports with test cases passing/failing, areas showing risky areas,
    • metrics showing trends across subsequent tests,
  • Tools to collect such feedback:

    • manual and automated test cases with test data,
    • tools and frameworks for automating tests,
    • tools for troubleshooting issues
share|improve this answer
add comment

Deliverables are dependent on the project, customer and product and should be agreed prior to the test preparation phase.

At the most basic level, a QA dept might product a single e-mail saying that the Product has passed or failed. On top of that, customers (or internal management) might require any level of detail, up to and including test scripts.

Strong V-Model projects obviously require greater detail and coverage in the documentation, although even the most agile of projects provide -something-.

When I'm defining the test deliverables I aim to provide the bare minimum, as any time spent providing documentation is time not spent testing. However, the two overriding concerns is that any omitted deliverable should not:

  • Introduce risk
  • Reduce customer confidence

As in test execution, if there is something is needed, it's provided.

I know that some might balk at my use of the words "bare minimum" but I've worked for companies where there was a lot of documentation provided that was then never looked at or even referred to. Product quality suffered as a consequence and that is a cardinal sin for a test department.

share|improve this answer
add comment

I would also suggest looking at things like

Return on Investment for automation Automation coverage (not code coverage but what % of your tests are automated)

Think bigger, make the QA team look more then just a team that produces metrics. -Erik

share|improve this answer
add comment

The deliverables from QA would be :

  1. Number of releases
  2. Number of internal iterations
  3. Delayed releases with possible causes would be great and nice to have
  4. Issues categorized as Client issues,Reopened issues ,Reappear issues(issues which were fixed earlier but have popped in again,This issue is different from regression),Regression issue
  5. Some figures like average bug/lines of code ,ex : There are 5 bugs reported on average for 1000 lines of code
  6. The most important point is if you back up this report with problems/challenges you faced during release ex : communication/Coordination with developer/Onsite team with some solution as a sync up daily call etc(Just suggestion) would help you and your organization in QA process
share|improve this answer
add comment

There are lot of answers for this question and you can find very good answers here! This topic is not easy stuff.

The QA team delivers the trust in the product and the feeling of safety it works and it works properly. On the other hand, the QA team provides information about the defects in the product, so that the organization is able to provide information for the customer service about the possible defects and workarounds.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.