Can testers benefit from knowing the status of unit tests?

Question

If a unit test passes, what you can infer is - barred exceptions - that the unit tested works as intended. Nothing can be said about all other components involved since they might be (and usually are) faked.

Testers' confidence in the quality of a product, on the other end, is boosted by looking at acceptance tests. Such tests operate at a higher level and should cover the behaviour of the system when used by the stakeholder.

The question: Say you, as a tester, are about to release a new version of the software. You can choose whether to look at the status of the unit tests - which cover pretty extensively the system, unit-wise - or not.

Does looking at the tests have any effect on you? For instance: - how is your confidence in the quality of the release affected? - would you test differently (different approach, more in-depth testing of some component with respect to others, ...) if you didn't check the status of those tests?

Answer 1

Here are a few considerations:

There are multiple reasons why a unit test might fail.

Unit tests have to be maintained just like the system under test does. When a unit test breaks, it might point to a bug in the system under test, or it might point to a bug in the unit test (e.g. a timing issue), or it might even point to a unit test that no longer makes sense.

No one here can tell you what your process should be.

There is no universal right answer for your question because it depends on the circumstances. Development and test processes vary from one company to the next. If I were in you, I would discuss it with your team and figure out what works for everyone involved.

You and your developers should agree on how unit tests fit into your development process.

If you aren't sure whether a failing unit test is a problem, you need to talk to your developers (or your manager, or their manager) about it. For example, you might agree that all unit tests need to pass before you start testing -- although that policy could have unintended consequences. Or you might agree there should be a bug opened for every unit test that's broken at the beginning of the QA cycle so that broken unit tests can be triaged like any other kind of bug. Or you might decide that unit test failures are an internal developer issue that has no relevance to testing because the developers are mature enough to fix the tests that really matter.

Answer 2

On some projects testers certainly can benefit from knowing the status of unit tests.

1) To initiate discussions. If unit tests have not passed before the testing / release, I would like to know why.

2) On some cases testers being able to see the status of unit tests and ask these questions has positive impact on maintaining unit tests. Sometimes testers not only ask the status of unit tests, but also what do they actually test.

3) On some projects the information what has been changed in the code/functionality is not automatically accessible to testers. Seeing what unit tests have failed at some point give more information about possible faulty areas in the software, allowing testers to concentrate their efforts better.

Answer 3

I had a really bad experience of negligence of unit testing once. I was testing this web based project which was extremely time critical(I had only like 10 days to test like 7 months of development), and after 2 days of testing I could not see a single test pass. Everything I threw at the project failed to pass. It turned out that every single field in all the forms failed validations - as simple as the email address format. When I had a talk with the dev team lead it turned out that unit testing had not been done, and to top it an excuse was thrown at me that the devs were trainees so we can't expect that from them! So first of all I was exploring 7 months of development for which I had 10 days which also included bug fixing and retesting and not a single feature of the project could pass for delivery. That frustrated me like hell! So yes it is very helpful to see good unit testing done!

Answer 4

I am system/integration tester working with competent developers. They are expected (and I fully trust them to do it) run unit tests before commit, and fix any broken unit tests promptly, and especially before start of any integration/system tests. We have huge unit test suite and on occasion some more obscure unit tests are broken for few days, if it does not influence core system testing.

Knowing that some unit test for core functionality are failing would inform me not to waste time on any system tests - there is no way to get valid results. For some specific values of "core functionality" and "valid results".

If unit tests of core functionality routinely fail and remain unfixed for prolonged time interval, it informs me that development process is broken. Not sure how beneficial this info would be to testers. Could be useful to management tho. In our company, manager of developers is different from manager of test team.

As a rule, it makes sense to measure what you can manage. There is little use to collect information about aspects of your system which are 100% out of your influence.

In your hypothetical scenario: If core unit tests are failing, we would advise management to delay the release. But developers know that, and would never dream pushing release with broken unit tests.

To release or not is managerial decision. "A" in QA does not stand for Assurance but Assistance. QA provides information so management can make informed decisions. If you have to ask this question, your process is broken, and no amount of "assurance" can mend it.

Answer 5

If I am a tester already involved in the same project, then I will find unit tests useful, in case they have ample evidence that they are run and executed concisely (description of expexted result and attached screenshots). Otherwise, if I have been assigned a responsibility to cover the quality requiremenrs of a project, then I would not be cinfident enough in case unit tests have no evidence of being run. Also, as a new tester in the project, I find it my responsibility to run scenarios in the process of understanding how the system works.