I recently came across a Google Tech Talk during which the speakers seemed to stress that trust in a test suite is crucial to the overall software development process. Here the link.

Can anyone share their opinion on this? If you agree that trust in a test suite is important, what do you do in your team to build that trust? I thought of asking the devs to sit down & basically code review whatever tests I wrote for the features they originally wrote. However, I'm not sure if that's the best way.




Trust in a test suite ultimately comes down to a few things:

  • does it miss important problems it should have caught?
  • does it report errors that aren't really errors?
  • is it brittle?
  • does it report useful information? (this is the most important factor, in my view).

Dev code reviews can help, but they're probably no better or worse than in-team code reviews - part of what you're looking for with this is to have an automated test suite that runs regularly, is reliable, and reports useful information. Exactly what constitutes these factors will vary depending on the organization and the team.

Some things you might want to consider:

  • how hard/easy is it to add new tests? Can you add new tests to your regression suites every time a bug slips out to the customers without needing to spend ridiculous amounts of time coding? (if you can do it with a few lines of data in a test data file, more power to you)
  • how hard/easy is it to update the tests when the application GUI changes? If you're effectively rebuilding your tests every time someone changes the tab order, you probably won't have a high trust test suite.
  • how hard/easy is it to update the tests when the process flow through the application changes? If it can be done quickly and easily, chances are your test suite is going to be more trusted than if it's a major rewrite.
  • what data is your test suite communicating and to whom? Some things to consider outside the number of tests run/passed/failed are known failures (where something is reported but hasn't been fixed yet), coverage statistics (which may or may not be useful to your application, depending), an overall summary of tests run/passed/failed over time (an ever-growing number of tests which mostly pass tends to build confidence).
  • What data are you communicating and to whom? No matter how good your test automation is, people are going to look at you (and/or your team, depending on the structure of your company) first, and impute trust based on how much they trust the information you give them.

None of this is "you must do this" - it's food for thought. Any method of building trust you come up with will be unique to you, your team, and your company.

  • 2
    One thing I have done is when filing defects having a "Found By" field where you can select Automation. Having some metrics that show that X defects were found by the automation can also help build confidence.
    – Sam Woods
    Aug 28 '13 at 15:45
  • Oh, absolutely. I'm accustomed to flagging defects found by automation with AUTOMATED SCRIPT - which also makes it easy to track the defects that automation finds.
    – Kate Paulk
    Aug 29 '13 at 11:28
  • This is a good idea, but in my current team we rarely get as far as raising a defect from our GUI automated tests - if our tests catch a regression we tend to just fix it immediately. For two reasons - 1) if it's just broken, then it just makes sense for the dev who broke it to amend his/her code straightaway, and 2) our test coverage for legacy areas is focused on the really important areas, so we know we can't release with a bug so we might as well fix it now. So you do need to point out that a benefit of failing fast is lowering the cost (i.e. some bugs get fixed w/o overhead of report).
    – testerab
    Aug 31 '13 at 18:34
  • @testerab - that's an excellent reason to work that way - perhaps you could flag commenting those fixes in some way so you can do a diff on the codebase from one version to another and use the flagged comments to indicate the number of bugs that were found and fixed immediately (it's a crude metric but gives an idea of cost savings)
    – Kate Paulk
    Sep 1 '13 at 19:19
  • Thanks @KatePaulk - that sounds like a neat low-cost alternative.
    – testerab
    Sep 2 '13 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.