How much should a tester understand and rely on implementation details?

Question

I am a test manager in my organization. We do mostly manual tests. Before that I was a developer. I find that my understanding of software implementation and architecture is very important for my role. Since I know how the architecture works, how the various components affect each other and in which areas to expect which bugs, I understand better where are the riskier areas and what to test and what not.

However perhaps I rely too much on the implementation details, and I tend not to test places where I think the implementation has lower risk of failing. But what if the implementation changes? Perhaps a more black box approach is in order.

Also, most testers in my group don't have development experience, so often they test the same module in different locations, even though if you are familiar with the code you understand that its the same module and its very unlikely that it will behave differently in different places. Often they don't test timing or performance issues because they don't understand how multithreading or memory leakages can cause bugs. There are plenty of examples like this.

So I have two questions

1. Should testers understand the code?
2. How much should testers rely on implementation details?

Answer 1

Short answers: 1. Yes they should understand the structure, 2. It depends.

Long answer:

If you ask a software tester why they are testing a specific function or decided to use a specific test technique/approach, they will probably give you the Willy Sutton answer: "Because that is where the defects are."

Of course that is going to be different for each organization, but in general the testers should take a "trust but verify" attitude about assurances and assumptions from anyone (not just developers or managers). If someone told me that the application will work the same in five different areas of the application, I will immediately run a fast smoke test to verify the assumption and then plan my approach based on it.

The primary concern is context. Even though the "intent" of the implementation is the same in each area of the application, the actual "experience" may be different. One simple example would be a Java application that implements a Java tree rule that always sorts the tree data in alpha order. However, one area of the tree lists recently accessed files, such that inverse chrono order is more appropriate. If the testers didn't check the assumptions in that implementation, that defect may never have been caught.

Answer 2

Should testers understand the code?

It depends.

How much should testers rely on implementation details?

It depends.

Why it depends? There are both pros and cons of understanding the code and architecture. When we interviewed candidates for a tester, we asked them to draw an architecture of a system they tested.

We did so because:

• We believed that such knowledge demonstrates the candidate can isolate or narrow down the root cause of a problem to a specific tier or module in the system. The knowledge can range from ability to inspect source code to general understanding of how and what information flows between different tiers. The first can help in remote debugging of the bug, the latter can be useful to report messages flowing between components, when writing a bug report.

• Knowing the architecture helps also test application at different levels: end-to-end with external systems included, just your system, only backend, etc., not just on front-end.

• For test automation engineer, understanding application APIs and structure is mandatory to plug a test framework into the system under test (test drivers, mocking, etc.).

On the other side, I learned, there are certain trade-offs when you focus on internals of the application too much. I prefer technical part of my job, so sometimes I overlook the perspective of an end user. I have also limited knowledge of my industry and the business, comparing to other testers and business analysts, that worked in the industry before becoming testers. They often can tell me whether my test case is realistic from our business point of view, or whether it is probable on production. Many of such business-oriented testers are also good in finding business edge cases, which tend to be different from technical edge cases that you mention.

Concluding, having both gray-box testers and black-box testers on board can be beneficial to your project and your team.

Answer 3

As others have said, it depends.

Testers do not need to understand code to be good testers. Testers who do understand code can use this knowledge to their advantage, but this does not automatically make them better testers than testers who don't understand code.

Your example of testing the same module in different locations misses the context: the base code of the module may be no different, but the pathways into that module are different, which introduces the possibility that one pathway may not be passing the correct information (or sufficient information) to the module.

For instance, say you have a form to collect or display contact information. The form can be invoked from a completed order, during the order process, during the invoicing process, and from the contact management module. You can create a contact during the order process and from the contact management module. This simple set of rules means it's necessary to test that you can't create a contact from a completed order or during the invoicing process. It also means you need to make sure that when you create a contact during the order process that contact is correctly associated to the order. In addition, you need to make sure that in all four scenarios, all the information you need is available when you view a contact, and that modifying the contact is available when your business rules dictate. In all these situations, the underlying code is no different, but the pathways into the code and the business rules around it vary.

Timing, performance, and threading issues tend to be more difficult to test manually anyway - issues like that are more likely to show up under automated testing and load testing, which tend to be their own specialties. Typically I'll be alerted to a potential timing/performance/threading issue by automated tests, and I'll do some manual investigation to dig further and try to force the issue - but I may not be able to because in some cases it's simply not possible to force something without having the application code running with breakpoints.

Don't underestimate your manual testers. The amount of work they do with the application means they have a good idea where problems are likely to occur, and they will cover this in their planning. They're used to the idea that they need to cover as many decision pathways through the application as they can in the time they have available. It may look to a developer like they're retesting the same code repeatedly (they are), but they're testing it through different pathways. They know that bugs happen in the communication between modules as well as in the modules themselves.

Answer 4

There are many ways to answer your question. :)

When I start testing something, I always initially approach it from a "black-box" perspective. From a black box perspective testers should not know or imply anything about the internal implementation of the software.

However, if I do know something about the implementation, then I may approach the problem from a gray box perspective. Gray box testing may help me target specific tests, but may also cause some bias in my tests.

If the goal of a test is to improve test coverage or code coverage then I would use a white/glass box testing approach where it would be essential for me to understand the implementation details in order to design effective tests more efficiently.

Now...the other question is whether testers should understand code/programming language. IMHO, yes they should understand the systems they are working on. Do they have to be proficient as developers...probably not. But, asking whether testers should understand programming concepts and fundamentals is like asking if nurses should understand first aid and common medical terms and concepts.

Answer 5

It depends what do you expect from tester and what kind of tasks. You should understand that test engineer which care about users and understand code, could find narrow places in code costs much more :) I believe it's desirable that manual test engineer can read code but not necessary. But if you mean automation test engineer or common QA engineer (manual and sometimes automation) it's necessary though to understand project archicture and read the code.