Signal to noise ratio during early testing

Question

The Context

For the past several years our QA team has been very active in manual and automated testing of our products which led to a very small amount of real-user complaints about the functioning and usability of our major products.

One of the things that has been working for us, is "start testing early" - while a feature is either partially implemented, or just released in a very immature state (in a "dev" environment). The idea was to give developers feedback as quickly as possible. This is motivated by the Barry Boehm's law: the earlier a problem is found, cheaper it is to solve.

The Problem

While it proved to work for us, there are related issues. Mostly, it is "Signal to noise" ratio.

On one hand, testing a feature that is not yet ready often helped to reveal critical problems, issues in designing/architecting a feature.

On the other hand, a lot of issues were not really bugs or problems, but rather things that were just "work-in-progress" or temporarily broken. This led to wasting time of both our developers and testers.

What would you recommend to improve on "signal to noise" ratio?

Answer 1

1. Greater communication between dev and qa so that qa already knows about wip issues.
2. Communicate first in person where there is disagreement, then document the agreement.
3. Clear definition on what quality metrics are being used at your company.
4. More discussion and planning up-front before the first line of code is written.
5. Talk about what testing is appropriate for an idea vs prototype vs first draft vs final product
6. Regularly scheduled review of inappropriately documented issues to learn and change
7. Agreed on ways for developers to mark things that are 'wip', perhaps in a special area in the ticket management system so that QA can see that first before they start testing.
8. Continue your general direction and make sure developer are on board with your 'shift-left' philosophy. My personal experience is that developers are always enthused to hear about it.

Answer 2

The paradox of when to write a bug...

Noise is merely a perception of those receiving something unexpected. If we look at ATDD and BDD the idea is that all tests are written before the coding cycle and all tests are considered failed. All Bugs are the results of a failed test and are usually only written after the coding has been delivered. But in reality a bug should have no negative connotations because it's just a record of behavior. Each bug can be either valid or invalid.

The Bug queue then becomes a behavior oriented database which, if managed correctly, it can become extremely valuable down-the-line. The main reason is in determining the most valuable Bugs to include in each iteration.

I worked a project once where we tested as early as possible, it created a ton of bugs which the developers didn't like. They said "We're not even done with the project yet?" It did add overhead to keeping track of things, but when the product was released it sailed right into production with not a single Severity one issue. The product owners became glued to our bug queue and wouldn't release the project until all Severity 1 and 2 bugs were done!

Today, most development teams are not required to show the Test Results from their own Unit Testing prior to moving onto the next phase. If that were done more often, there'd be a lot better endings. Unit Testing is the best place to test, even for the QA team.

Moving forward, QA Team members are going to have to pick up programming skills to stay relevant. Automation isn't only about approaching from the GUI, it's about approaching from the code side.

Answer 3

I'd argue that you're kind of wrong... You're not wasting you're testers time by finding these issues. As you don't elaborate as to what exactly is wasting your time, I'll make a dangerous assumption in thinking that it's the communication of the noise amongst the actual issues.

First, I'd like to re-visit my previous point; having testers find and take note of these issues (and I mean take note, not engage in any formal bug reporting procedure, which is a huge waste of time in this case) is inherently a good thing. It may not lead to the direct increase in value for your product, but it will increase the skill of the testers. If these notes are legible by an outside party, which in clean testing is a primary goal, then they alone may be suitable enough to communicate to the developers what the issues are. Good testers should try and evaluate the potential noise-factor (a broken button vs. incorrect representation of data) and highlight appropriately.

If a more engaged approach is needed, a timeboxed paired testing activity with devs and testers can work wonders for cutting the doodie.

Formality and time wasting procedures are dead weight in such a rapidly changing environment.

Answer 4

Changes in backend are easier (because from user's POV there are no changes, or defined changes). Trickier is how to early test UI changes, especialy with radical changes in UI.

Instead of formal round of early testing, we had great success with developers presenting new redesigned page while it is still in DEV to wider audience: 5-10 people. Developers does all the clicking (and can limit access to parts which are broken), but users can see how page works and feels. This feeling is much harder to get from just reading page design docs.

Most valuable audience we found are trainers. They know our system well, but know also what parts are trickier or less intuitive to users. But anyone (representing future users) is welcome on such presentation.

Developers can show the new GUI, and get feedback early (can write list of issues in their own terminology), without creating too many bugs and extra work triaging them.

Communication instead of documentation.

Answer 5

I've worked on cross-functional agile teams, and the simple answer is if the issue cannot be resolved the same day it was identified, then write a defect.

Keep a whiteboard / parking lot space to document all the stuff found during the day. At the end of the day, anything that hasn't been resolved needs to be documented as a defect.

Do you have different priority or severity values you assign to defects? Why can't QA just mark items as "low", and you simply ignore or filter out those issues until you have time to address them?

How you decide what to rate a defect is another issue to talk about. Do you have definitions for what qualifies as a Low/Medium/High defect? Do you and your QA team have an agreement on what the repair timeline is for a defect that is a High priority vs a Low priority? Do defect repairs only come in complete releases or is everything fixed on the fly? If it's fixed on the fly how do you identify all the regression tests needed for an eventual release?