The Context

For the past several years our QA team has been very active in manual and automated testing of our products which led to a very small amount of real-user complaints about the functioning and usability of our major products.

One of the things that has been working for us, is "start testing early" - while a feature is either partially implemented, or just released in a very immature state (in a "dev" environment). The idea was to give developers feedback as quickly as possible. This is motivated by the Barry Boehm's law: the earlier a problem is found, cheaper it is to solve.

The Problem

While it proved to work for us, there are related issues. Mostly, it is "Signal to noise" ratio.

On one hand, testing a feature that is not yet ready often helped to reveal critical problems, issues in designing/architecting a feature.

On the other hand, a lot of issues were not really bugs or problems, but rather things that were just "work-in-progress" or temporarily broken. This led to wasting time of both our developers and testers.

What would you recommend to improve on "signal to noise" ratio?

  • 5
    I am very happy that such relevant questions are being asked on this forum. So different from our usual diet... Jun 5, 2017 at 21:35
  • 2
    yeah, 2nd great question from @alecxe ! Jun 5, 2017 at 22:53
  • testing a feature that is not yet ready often helped to reveal critical problems And these problems weren't caught by the developers? Jun 6, 2017 at 5:49
  • @JohnGordon absolutely, some of them do get caught by developers as well, some others are caught by the QA team - we basically have white, gray and black boxes testing from the very beginning of a feature. Also, one of the other problems we are trying to solve with early QA team involvement is to compensate for the lack of unit-testing on the UI side. This is not quite right and we should improve on that, I understand. Thanks.
    – alecxe
    Jun 6, 2017 at 12:04

5 Answers 5

  1. Greater communication between dev and qa so that qa already knows about wip issues.
  2. Communicate first in person where there is disagreement, then document the agreement.
  3. Clear definition on what quality metrics are being used at your company.
  4. More discussion and planning up-front before the first line of code is written.
  5. Talk about what testing is appropriate for an idea vs prototype vs first draft vs final product
  6. Regularly scheduled review of inappropriately documented issues to learn and change
  7. Agreed on ways for developers to mark things that are 'wip', perhaps in a special area in the ticket management system so that QA can see that first before they start testing.
  8. Continue your general direction and make sure developer are on board with your 'shift-left' philosophy. My personal experience is that developers are always enthused to hear about it.
  • "Communication" was the first idea we've discussed when we've talked about it during a dev meeting. After that, it took us 45 minutes to argue about how exactly are devs gonna communicate WIP things to QA. Decided to stick with a wiki page for the current release with a special paragraph about WIP parts for now. It probably does not scale well though, but we'll see if it works for us. Thanks!
    – alecxe
    Jun 5, 2017 at 18:40
  • I recommend 1:1 conversations and then document. If it took 45 minutes for the initial conversation, then there are some issues that need to be worked through Delegating out to a document won't be effect in solving underlying issues in my experience. Jun 5, 2017 at 22:48

The paradox of when to write a bug...

Noise is merely a perception of those receiving something unexpected. If we look at ATDD and BDD the idea is that all tests are written before the coding cycle and all tests are considered failed. All Bugs are the results of a failed test and are usually only written after the coding has been delivered. But in reality a bug should have no negative connotations because it's just a record of behavior. Each bug can be either valid or invalid.

The Bug queue then becomes a behavior oriented database which, if managed correctly, it can become extremely valuable down-the-line. The main reason is in determining the most valuable Bugs to include in each iteration.

I worked a project once where we tested as early as possible, it created a ton of bugs which the developers didn't like. They said "We're not even done with the project yet?" It did add overhead to keeping track of things, but when the product was released it sailed right into production with not a single Severity one issue. The product owners became glued to our bug queue and wouldn't release the project until all Severity 1 and 2 bugs were done!

Today, most development teams are not required to show the Test Results from their own Unit Testing prior to moving onto the next phase. If that were done more often, there'd be a lot better endings. Unit Testing is the best place to test, even for the QA team.

Moving forward, QA Team members are going to have to pick up programming skills to stay relevant. Automation isn't only about approaching from the GUI, it's about approaching from the code side.

  • Absolutely, we had some arguments with dev team about these kind of early bugs! Good thoughts! I did not mention that even though this was probably relevant: another motivation to do the QA testing early was that our UI devs were not unit-testing their code at all. We try to compensate with more e2e testing during the whole life-cycle of the application up until it is in prod..I know, it is sad. Thanks so much!
    – alecxe
    Jun 5, 2017 at 18:38
  • 1
    Too often I hear from the Devs we don't have enough time to Unit Test, and they are right! The root of this problem is in the planning of the iteration. If developers don't have enough time to test, then the due dates are wrong! Product owners and Users want thing right now which doesn't help because they are the ones footing the bill. The only way out of this mess is 2 week iterations that continually deliver what is the most value to the product. QA Teams can lag each iteration but that's not optimal. Just shift left as they say... Jun 5, 2017 at 22:04

I'd argue that you're kind of wrong... You're not wasting you're testers time by finding these issues. As you don't elaborate as to what exactly is wasting your time, I'll make a dangerous assumption in thinking that it's the communication of the noise amongst the actual issues.

First, I'd like to re-visit my previous point; having testers find and take note of these issues (and I mean take note, not engage in any formal bug reporting procedure, which is a huge waste of time in this case) is inherently a good thing. It may not lead to the direct increase in value for your product, but it will increase the skill of the testers. If these notes are legible by an outside party, which in clean testing is a primary goal, then they alone may be suitable enough to communicate to the developers what the issues are. Good testers should try and evaluate the potential noise-factor (a broken button vs. incorrect representation of data) and highlight appropriately.

If a more engaged approach is needed, a timeboxed paired testing activity with devs and testers can work wonders for cutting the doodie.

Formality and time wasting procedures are dead weight in such a rapidly changing environment.


Changes in backend are easier (because from user's POV there are no changes, or defined changes). Trickier is how to early test UI changes, especialy with radical changes in UI.

Instead of formal round of early testing, we had great success with developers presenting new redesigned page while it is still in DEV to wider audience: 5-10 people. Developers does all the clicking (and can limit access to parts which are broken), but users can see how page works and feels. This feeling is much harder to get from just reading page design docs.

Most valuable audience we found are trainers. They know our system well, but know also what parts are trickier or less intuitive to users. But anyone (representing future users) is welcome on such presentation.

Developers can show the new GUI, and get feedback early (can write list of issues in their own terminology), without creating too many bugs and extra work triaging them.

Communication instead of documentation.


I've worked on cross-functional agile teams, and the simple answer is if the issue cannot be resolved the same day it was identified, then write a defect.

Keep a whiteboard / parking lot space to document all the stuff found during the day. At the end of the day, anything that hasn't been resolved needs to be documented as a defect.

Do you have different priority or severity values you assign to defects? Why can't QA just mark items as "low", and you simply ignore or filter out those issues until you have time to address them?

How you decide what to rate a defect is another issue to talk about. Do you have definitions for what qualifies as a Low/Medium/High defect? Do you and your QA team have an agreement on what the repair timeline is for a defect that is a High priority vs a Low priority? Do defect repairs only come in complete releases or is everything fixed on the fly? If it's fixed on the fly how do you identify all the regression tests needed for an eventual release?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.