8

Sometimes, tasked with testing a piece of software for which there are no requirements, a tester will resort to white box testing. Are there times when that is inappropriate?

Here is an example. At a prior job, we had some loosely-worded requirements for some reports and a general requirement for a datamart (a reporting database) for running ad-hoc reports. The developer chose to implement the (loosely-specified) reports on top of the datamart.

The datamart had a different schema from our transactional database. There was a tool for transferring data from the transaction database into the reporting database. The tool ran on a server and was not user-facing. The tester was tasked with testing the tool, not the reports.

The tool was driven off of SQL queries. The tester's strategy was to read each query, figure out what it was supposed to do, and write a set of tests that tested the behavior of that query. So for example, say the goal of a query was to read rows from table X in the transaction database, fiddle with the data, and write the results to table Y in the report database. The tester would write a test that inserted some rows in table X, calculated what the expected results should be, ran the tool, and then verified that the expected rows showed up in table Y.

I always wondered what would happen if a test failed: would the tester question whether there was a problem with the tool, or would he assume his test was wrong? After all, the tool had no explicit requirements. This seems like a risk for any type of white box testing, and I wonder whether the maintenance costs of this kind of test justify the effort.

3
  • 2
    I'm not really clear what your question is: this doesn't seem to be a problem with white box testing, but with lack of a reliable test oracle. Why couldn't the tester ask the product manager whether a failure was a problem? Why couldn't the tester ask the product manager for more information about the purpose of the reports, and what data was key to them?
    – testerab
    May 17, 2011 at 14:56
  • @testerab Thanks. I updated the text to reflect that there was also a requirement for ad-hoc reports, and that the tester was tasked specifically with testing the datamart, not the reports.
    – user246
    May 17, 2011 at 16:34
  • The problem here is that the dev might have got the wrong specification and developed the wrong piece of software. Then, when a Tester actually finds a bug, there is a chance for the dev to say that 'it is not a bug' Dec 12, 2016 at 14:56

3 Answers 3

4

Excellent question. I think this comes under the heading of the "Application is its own specification". In the absence of requirements or specs, the best way of knowing what an application is supposed to do is to start using the application. If you actually have access to the code of the application, that extends this idea where the code itself becomes the requirements.

In the example you gave, if the test failed, that's when a communication channel to the developer needs to be opened, if it's not opened already. Communicate to the developer what the assumptions were, what was done, and what the resulting output was. Dialog with the developer then can determine if the tool is functioning as expected or if this is a found "bug".

I think, to answer the original question, in the absence of clear requirements, white-box testing is a more than appropriate tool to use if the internals of the AUT are available. It really is the only way to know what test cases need to be run.

1
  • 2
    I agree wholeheartedly with advising the developer the specific tests that were done. We rarely have requirements so I do white box testing quite a bit. When testing a feature, when I have finished 100% of the scenarios that I came up with, I e-mail the developer all the test scenarios (& pass/fail results) I did, and ask if they had any test scenarios in mind that I did not cover. We have an excellent team with great communication skills, and this has worked well for us. May 17, 2011 at 18:40
9

The problem here isn't white-box testing. The problem is that there is no specification. This is where domain knowledge and product familiarity really matters. A tester who just tests that the code does what it was intended to do (from the dev's point of view) by reading the API could very well miss a variety of issues, such as a business requirement that wasn't communicated to the developer.

When I run into this situation, I do three things in parallel with white box testing:

  1. I ask around to ensure I didn't miss any specifications that were written
  2. I raise concerns up the ladder - dev, PM, business owner usually.
  3. I ask many questions, and ask them visibly. Our team puts an "Open Questions" section at the bottom of our wiki specifications for precisely these situations, and I can then email out a link to that section of the functional spec the developer has created to everyone who might know about "hidden requirements". If the dev didn't create a spec, I'd make my own document that included an "Open Questions" section.

The risk of white-box testing in this scenario is that I could more easily neglect these steps if I weren't self-disciplined or experienced enough to know better and miss business requirements that weren't successfully communicated to the developer. I will also need to defend to the developer and PM why I need to do these steps even though I can "just look at the code", but that usually isn't difficult. If I were only black-box testing, I wouldn't be able to make any progress at all until these steps were done, so there is very little risk of these steps being neglected.

Still, this isn't a reason not to white-box test. The specific advantage of white box testing in this particular case is that I can make more progress on my tests before I get complete answers to what the product should actually be. This is in addition to the other advantages of white-box testing, such as finding features that the developer knows to implement that the business owner didn't know to ask for (e.g., monitoring and logging usually fall into this category at our company).

2
  • 2
    Good answer, Ethel! I would add that there is never a case where there are no requirements. Requirements exist, even if not explicitly written. I approach such situations this way: strazzere.blogspot.com/2010/04/… May 24, 2011 at 18:05
  • Nice blog post :) Thanks for the comment, it's a good point! May 24, 2011 at 18:59
0

Why not test the final reports? This will make defining expected results easier and verify your ETL tool as well. Obviously, for a certain trade-off which I discuss later.

The problem I found with testing only an ETL tool in isolation is that a datamart DB schema (e.g., star schema) is often more cumbersome to read and understand than the final report (this is for query performance reasons). Consequently, assertions/checks for data in datamart are harder to describe, read, and maintain and because of that more error-prone. Tests that will verify the final reports will be easier to understand and also deciding what is the expected result will be easier, because business people or a customer will be able to answer that.

Trade-off. This approach has a number of obvious shortcomings related to testing more layers at once without testing them prior in isolation:

  1. Isolating whether the error is in ETL tool or in the report definition.
  2. Report definition errors may mask errors in ETL tool.
  3. Feedback is delayed.
  4. Text execution is longer.
  5. Finally, automating tests for reports is usually harder (tables, graphs), because you need to parse the report document.

We were able to limit some of those problems. For instance, we have limited the first problem by slowly including assertions on datamart into end-to-end tests. First, we had an end-to-end tests that checks only report, and as we learned about the relation between original DB and datamart DB, and between datamart DB and the report, we were able include more specific checkpoints in the test. Now, when the test fails, we are able to say in which layer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.