Can knowing too much about the tested code be a disadvantage?

Question

Shift left testing is an approach in software testing to involve testers into software lifecycle as early as possible. I always understood this as helping in finding conflicting requirements, but also reviewing developer's tests and implementing integration tests (tests verifying integration of single components) before end-to-end tests (that are run against the whole system).

However, I am afraid that when I am getting into the internals of the system under test, I am losing wider perspective of an end-user. Sure, knowing the architecture helps in writing gray-box tests, but I am no longer having user perspective or time to get this perspective. Quoting "Lessons Learned in Software Testing: A Context-Driven Approach":

If your primary focus is on the source code and tests you can derive from the source code, you will be covering ground the programmer has probably covered already, and with less knowledge of that code than she had. […] The advantage of black box testing is that you probably think differently than the programmer, and thus, are likely to anticipate risks that the programmer missed.

How to be a technical tester and preserve the end-user perspective at the same time?

Answer 1

How to be a technical tester and preserve end-user perspective at the same time?

I believe that there is no 'fix' to this other than to become proficient in both areas and to combine those skills through actual work. The two different aspects are not a binary choice of one or the other. I would focus on the specific things to get good at in each area and allow the combination of the skills to come together through your work.

So the skills I would try to advance in each of the aspects is

Technical Tester - unit tests, mock and stub, integrated tests, programming skills, writing DRY code, separation of concerns, isolating application unit under test, DRY code, etc.

End User Perspective Tester - learn the business domain, learn current business objectives, learn how/where revenue is earned, understand reasons for new and changed features from business perspective, understand business goals for upcoming quarters, learn about customer needs, understand customer viewpoints, pain points, study feedback surveys, obtain customer service feedback, etc.

In terms of your main title Can knowing too much about the tested code be a disadvantage? I would say yes if you focus too much only on the technical aspects above and not enough attention to end-user perspectives

To do this well means learning how to use different personas and perspectives and how 'wearing a different hat' works. This skill takes some practice and takes time to learn.

At the end of the day you cannot 'avoid' technical knowledge affecting the end user perspective to some extent. The goal should be to be aware of that fact and to make sure that you take extra effort to make sure the end user perspective is still listened to. This can be done through a number of activities including involving others and paying more attention to customer feedback.

In addition to the above, one approach that I use that it somewhat in between the 'black box' (often user perspective is black box) and 'white box' (often meaning having knowledge about the code itself) approaches is 'grey box'. In this case one doesn't know or have access to the code internals but you do have access to control state. A couple of examples of this:

• The ability to set or seed a database appropriately for a test to run
• The ability to control a web session and visit a specific page of a multi-form flow

Many experienced automation developers have found the above capabilities can be critical to actually having reliable test suites that run quickly and add huge value.

You also need to both acknowledge and take advantage of what the automation does not do in terms of addressing usability and still do real user manual testing for that perspective, but make sure the testers have a good understanding of what the automation does and the value it already provides and how their input should address different aspects and which aspects they should be.

Answer 2

Knowing more information is never a disadvantage when you take decisions.

The question is how well you can classify, aggregate, structure and order that information.

Basically not to miss end-user view a tester should start to design the tests from the business and functional specs when the code is not ready yet (ideally there should be a segregation of duties and automation tester should only code the tests however in the real life if often happens that "automation testers" are involved in all types of QA activities).

So having the tests which design is coming from the end-user facing specifications you now can dive deep into the code since you already have the end-user tests in place and you are free to add the tricky tests using the knowledge of how your system is actually implemented.

So what you described is not actually a big problem if you start from the end-user side and then switch to code.

Answer 3

What I feel knowing the application code base can help the tester on which the application is built. I see most of the tester have a mindset like

"What do I have to do with the architecture, Base code of the application! or Why should I come to testing if I need to understand all the class and methods or I could have become a developer instead or tester."

For them, testing is all about break the system & report the bugs.

Nowadays applications are becoming more complex day by day, So as a tester we can't sit back & follow the same traditional QA practice, just like by keying some data through the user interface or some API tool & try to understand underlying components of the application & plan their testing so that we can identify hidden errors in the application. I don't think it going to work for a long run So it's a very good practice to understand the component beneath the hood with the code base from where application originates.

Answer 4

You can't see the forest through the trees.

The way I 'try' to solve it(to regain larger end-user perspective), is to step into end user's shoes and ask broader questions like ' What is end user's goal here and how the application is going to help in achieving it?'

This way I try to zoom in and out of the micro and macro views of the application as required.

Answer 5

Usability testing suffers from knowing the code. All the rest benefits. Knowing algorithmic edgecases is never a bad thing for unit, functional, performance, load testing. Knowing the code improves your knowledge about the area. And as for usability - just use targeted groups during acceptance test.