Actual data for why developers shouldn't be the only ones testing their code?

Question

Can anyone provide actual data or scientific studies that show developers are (or are not) less effective at testing their own code than "independent" testers? I've seen many opinions but little hard data.

EDIT: A few followup clarifications:

1. This is referring to system/e2e tests.
2. There are also two interpretations of "developers testing their own code": an individual developer testing the code they authored, or more generally developers testing code that may have been written by other developers. Data for either one would be instructive and I think there are different considerations for each. We currently do both.
3. We use team inspections in most cases for requirements, code, system/e2e tests, etc., with the exception of unit tests. In my opinion this does mitigate at least some of the risk of the test author being "blind" to gaps in what is being tested.

Background:

I'm part of a team of around 20 developers, and until recently we had no dedicated test team. On my first project I was the "software test lead" while still doing actual development, and the expectation was that when you implemented a feature, you didn't merge it until you had also written the tests related to it. We also have a very good culture of celebrating finding (and fixing) bugs, egoless reviews, etc., and to top it all off we have produced very high quality results. In my opinion, it is possible for developers to be just as enthusiastic about breaking their own code as independent testers can be, if you have that kind of culture in place.

I just started reading How Google Tests Software and it seems we aren't the only ones who have had good results when developers are responsible for testing their own code. I haven't even gotten past the introduction yet but the whole idea of having a "Engineering Productivity" group rather than a group labeled "QA" or "Testing", and those people working to enable developers to test faster and better actually sound very similar to my experience: though we didn't have separate titles there are people in our group that are known for writing tools to increase productivity, whether in testing or otherwise, which combined with the culture of quality mentioned above has proven successful.

Our management is pushing for creating a dedicated testing team. They've given a number of reasons, and I've seen many of these reasons/opinions offered in answers to questions like How does a tester's perspective towards software differ from a developer's? and Can developer do automation test for the feature that he has implemented?, but while I've seen lots of opinions (many from experience, and I do respect them as such), I've yet to see any hard evidence or data that developers are inherently less effective at testing their own code than an "independent" party. My biggest concern with moving from our current model to a "separate test team" model is that developers will no longer view quality as their responsibility, and I've seen some of this mentality start to set in on a project where we had a test team. It was an offshore team that proved utterly incapable of writing quality tests, but because we didn't ditch them until a year into the project, we built up a lot of quality debt while also allowing many devs to get out of the mindset of also being testers.

I'd like to push back on this idea of separating development and testing but want to do my homework first. There are opinions and anecdotal/experiential evidence that support both viewpoints, but I'm interested in knowing if there's any actual data to support one or the other (or "it depends").

Answer 1

There doesn't seem to be a lot of research data on this; this is what I found:

Waiting for builds:

This article about "Why software testers can't test" quotes a survey from IBM (The Future of Testing: Where Do Testers Spend Their Time? 2014):

59% of those polled, the largest percentage for all the available choices stated that the one activity they wish they could spend less time doing was…”waiting for test assets.” Dagger.

This has also been my experience as a manual tester in the past. It even gets worse as you find some critical defects that block your testing. You need to hand it back to the development team and wait for the next testable version.

Shortcuts:

I have read an article a while back (but can't find it anymore) that stated that as their company switched to a dedicated QA-team the overall quality of their product started to drop. As the developers thought the testers would test it thoroughly, the developers stopped most of their testing efforts. Their work got only a little bit worse because they didn't find obvious issues instantly. The testers were under more time pressure than the developers. They were lagging behind, while the company wanted to ship. The testers started taking larger and larger shortcuts, missing more and more important issues.

On a personal note, fixing issues found by a separate team are harder to fix due to context-switching and forgetting how that feature was built. This should be provable with data: there is a lot of research about multitasking and context-switching.

Companies are cutting their QA-teams:

• Yahoo
• Unknown

Dedicated QA-team:

I am all for a QA-group, as long the members are part of dedicated software development teams. Where they help with the whole team approach to quality and quality-build-in. Preferably helping the team to do test-first practices.

But having a dedicated test team that you hand over a testable product to... that feels like a horror I hope to never again experience in my life.

I love swarming. Swarming the requirements, swarming the architecture & design, swarming to create automated tests and code, swarming to refactor and then swarm exploratory testing sessions as well.

Training:

You can train developers to switch between build-thinking and critical-thinking. Having critique on the product is just another mindset. Most developers are smart enough to be able to master these skills. You just have to motivate them that it is important. I truly believe a very good developer must also be a very good tester.

Answer 2

Disclaimer: this answer is purely based on personal, anecdotal experience.

In my 10 years as a Software Developer, I have known 4 different situations:

• no tester (besides myself),
• tester in another team, large delay (weeks),
• tester in another team, short delay (hours),
• tester in same team, short delay (hours/days).

Note that Unit-Tests were always on my end, the tester instead would be in charge of high-level tests, either manual or automated.

Spoiler: the subjective evaluation of the quality of the delivery has informed the order in which I listed the situations.

---

Tester

First of all, in my experience having a second pair of eyes is invaluable:

• a different reading of the specifications/requirements help identify ambiguities (arising as the tester expects a different output than the developer),
• a different person simply thinks about different tests to do.

In short, as a developer, if I thought about a corner case, my code will handle and my tests will show case it. The issue, however, is about the corner cases I didn't think about.

Secondly, a professional tester will develop instincts of things that can go wrong and my (sole) data-point has been that they were much more vicious in the situations they came up with (a lot more variations than I had thought of).

Therefore, a professional tester is invaluable when it comes to raising the quality of a project.

---

Delay

Maybe simplistic, but my best experiences were when the tester would test quickly.

Context-switching is generally branded as lowering productivity, however it has been my experience that is also lowers quality.

The better recollection I have of the data-flow, specific concerns, etc... the more likely I am, when fixing a bug, to:

• avoid introducing another bug instead,
• think about potential other buggy situations and fix those too.

This means that a shorter delay between production and feedback not only increases productivity, but also quality.

---

Them vs Us

I have not had a problem with individuals, but I have regularly butted my head against different priorities.

When the Dev Lead and QA Lead disagree on the priorities, and this means nobody can test my delivery quickly, then... it introduces delay, and all its consequences.

It also engenders some degree of frustration; or even led to functionality being delivered in production before being reviewed by QA. I guess it could help keep developers on their toes... but I do not recommend it.

---

Involve

Overall, my best experience has been with a dedicated tester in the same team, and office.

By involving the tester from the beginning:

• the software is designed to facilitate testing,
• if necessary, testing-only requirements are incorporated in the design,
• being involved in the design, the tester knows the intended functionality and requirements (beyond the written specifications),
• being situated close-by, the tester and developer can ask for clarifications on the fly.

The latter part has been instrumental in raising the quality of the product I was working on.

If you can speak to the tester/analyst to ask for a clarification, you are likely to do so. If you can call them, you may very well do so. If it requires an e-mail... you might do so... but probably will not. Because you are writing the code now, and you are not going to pause until they read your e-mail and reply. This is the difference between clarifying something, or assuming "for now"... and we all know how that ends up.

The same goes the other way around. Rather than waiting on the implementation of the test because it's unclear how to use your code, it's much simpler for you to answer now. And the faster your reply, the less likely they are to put your testing on the back burner as they switch to another task.

Of course, interruptions can be annoying, etc... however, a once in a day clarification from one person is not the end of the world whereas a once in a day bug introduced is a significantly lower quality.

---

Conclusion

In the end, my experience is that the best quality results from having a dedicated knowledgeable tester:

• involved from start to finish,
• in close proximity,
• providing feedback in short order.

Having them in the team is not necessary, but helps on all points, and probably helps switching the "them vs us" mindset off.

Note: it does not necessarily mean having the same person to report to; my company at the time used project-teams which assembled people from different actual teams and put them together in the same office.

Answer 3

TL;DR: Devs write own unit tests. If you want different eyeballs to look at the unit tests code, code review is the answer.

EDIT: OP clarified question: it is about e2e tests. For e2e tests, using separate team of developers is possible but adds different kinds of challenges. Added to the bottom.

Related question: How does a tester's perspective towards software differ from a developer's?

Quality cannot be bought, but has to be paid for - no shortcuts

---

No studies, just common sense.

Developers should write their own unit tests (and fix the broken ones), because most recent developer knows the best the recent changes, and is in the best position to investigate the consequences.

It makes no sense to use different QA test team to write and fix unit tests, because they will have to learn what developer just learned and did (and devs have in operational memory), ask them many questions why (and possibly create misunderstandings). Just huge duplication of efforts, total waste of time. All those questions should be asked during the code review and resolved by original developer.

Separate QA testers (if used, a big if) are best positioned to write system/integration/e2e tests, because those tests may (or may not) benefit most from different set of eyes, possibly using different unsaid assumptions, and sometimes using different programming language: core devs might be more concerned about performance, while QA e2e testers are much more concerned about the productivity (if testing needs to run faster for valid business reasons, buy more servers and run test in parallel).

But often, devs are also writing system/e2e test, so they can fix any changes in the UI, because they are best familiar with the changes (and less surprising: original developer does not need to investigate what changed).

---

E2E tests are very different animal. With those, second pair of eyeballs might be beneficial.

Again, no studies, just real-life experience earned the hardest way: by own sweat and blood :-)

In our company, we use all three approaches:

• developers also writing e2e tests (maintaining tests written both FitNesse, old homegrown automation framework, and new test using new homegrown framework enhancing Selenium WebDriver)
• separate test automation coder (just one :-( ) catches up with e2e tests which were not developed during original development (business decided to take shortcuts)
• tests are developed using hybrid approach and pair programming.

Automation tester (me) developed in-house framework for Page Object testing, and helper functions using it, simplifying the code and increasing productivity. But e2e tests rely on convenient reliable locators (IDs and names) for all semantically significant page elements. Which is the root of the problem.

As you can imagine, developers do not like learning additional framework, even if it improves productivity when writing e2e tests.

OTOH, when backfilling tests, sometimes convenient locators are missing. Workaround is to use flaky XPath locators (with all the involved problems - better avoided), spend some more time to develop CSS locators, or ask developers to add convenient locators (names/IDs) to existing code.

Adding locators is easy, but scheduling it is hard. It may take hours or days (depending on developers workload, willingness to cooperate, and ability to switch context), and test have to wait for those new locators (and forces them to switch context while waiting for improved locators), so adding locators on demand approach is far from optimal.

Best approach (in our experience) is pair programming for e2e test development (pairing automation tester with developer) to develop page object for new pages and smoke test touching many/most widgets on the page. Automation tester knows the automation part and how to do it, developer knows how to add locator when needed. Using this approach, adding locator and using it in a test takes seconds, not hours or days.

Afterwards, automation tester can work separately without core developer to add more and more complex tests, reusing the locators.

But without this "paired" step, developing locators is exercise in frustration for automation tester:

• it either takes very long time waiting and swapping context,
• spends lots of time designing location strategies using CSS and other attributes which are available,
• or uses flaky XPath.

We are transitioning/settling/improving our process for pages using e2e test with Selenium WebDriver. Single-page application pages with Angular make it more complicated.

Angular and single-page app design pattern also opens yet another can of worms: who is responsible to fix e2e test when page fails after UI design change.

My feeling is that test should be split into (at least) two groups:

• smoke test, developed by pair programming as mentioned above. Test failures (changed locators) are fixed mostly by UI developers themselves.
• more complex tests, developed and maintained by QA test automation team.

Of course this requires very tight cooperation between developers and QA test automation.

In a "throw it over the wall" approach (pages are dumped on QA automation without decent locators and without the ability to add necessary locators quickly and conveniently), productivity of QA automation will be very low, tests will be flaky and unreliable, and pushback against e2e testing as waste of resources will be strong.

Add to it the temptation to use automation testers with lower-level skills, and disaster is inevitable, as we can see from questions asked on this forum daily, which are trivially answerable by better locators and trivial amount of programming skills.

Answer 4

Disclaimer: not hard data, but something from a personal experience.

We currently have a good representation of why developers should write unit-tests for the changes they made (at least, critical and most error-prone parts if under time-pressure, as we are).

One of our products is an AngularJS application, the front-end of which is being developed by several UI JavaScript developers. Our JS side is tens of thousands of lines of JavaScript without a single unit-test. "We don't have time for it" is one of the explanations UI developers might provide when asked why. In other words, our dev side is not participating in any testing activities.

Our QA team is trying to compensate for the lack of unit tests by adding more end-to-end browser automated tests (we currently have about a thousand of them), which basically flips the famous pyramid upside-down.

This has multiple consequences:

• we often have a manual testing period when we (QA team) find trivial issues which could be quite easily caught with unit-tests
• sometimes this kind of issues make it further to staging and production which makes them more expensive to fix
• end to end tests are really slow and flaky
• it is often required to research an end-to-end test failure to find the actual cause of it - the feedback loop is really slow
• we also have some problems communicating markup changes

Answer 5

From experience, I have found that developers are very effective at testing what the code does, but not as good at testing what the code is supposed to do.

The first example that popped into my mind is a GUI I designed. I tested the daylights out of it, and though it was flawless. Brought it in front of my customer, and 10 seconds in he had crashed the system. He was kind about it, but afterwards, I had to figure out why a system I had tested failed so miserably.

The answer was that in my testing, I'd click buttons. I'd do

• Click button 1 - make sure it does X
• Click button 2 - make sure it does Y
• Skip button 3 - I know it doesn't work yet
• Click button 4 - make sure it does Z

How the heck did I miss button 3, when I knew it was broken? Well, I did my testing so many times during the build that I developed a habit. I knew button 3 didn't need to work yet, so I kept skipping it. Then, when it needed to work... I kept skipping it.

There's plenty of approaches I could have done to resolve this (automated testing from the start would be one), but the anecdote highlights a key challenge for developer testing: we know what the code does. In our head is a giant mental model of everything that block of code is doing, and we subconsciously leverage that model during testing.

I think you can teach developers to do good testing, but I think you have to actively do so. It's hard to step back from your own model and approach software from a blank slate. That's why we have code reviews.

Answer 6

When working on something complex, there is always the risk of bias, blind spots, tunnel vision, and stress induced malfunction of the thinking parts of the brain. ( one good book, though from outside the IT world, is Processing Under Pressure, by Matthew J Sharps)

That is why it's always good to have another human around,
not a yes-person or naysayer, to have another look and guard your back. For code, but also for buying new paint for your house, buying a new suit or writing an important letter.

This requires an environment where this friend with the title * honorable enemy * can function without getting beheaded. Many multimilion dollar movies flopped because the director only accepted yes-persons to review the ideas.

It is possible let a colleague developer do the testing. Except a similar background and similar way of thinking can still lead to the same blindspots. Enters the tester, or test role.

Dangers: silo thinking, us vs them, throwing over the wall attitude. If developers see testers as 'failed developers', testers see developers as 'reckless cowboys', end users see both as 'geeks from Mars', end users get seen as 'those (dumb) end users', ... then nothing can save the product.

Testers as part of a team, with short communication lines, where everyone is interested and dedicated into creating quality, it can be done.

Answer 7

A few years ago I've done a quite extensive literature study into testing, especially regarding benefits of testing. I have not encountered a scientific study in precisely what you are asking (developer versus tester), but I did encounter a quite interesting scientific study: Mäntylä, M. V. & Itkonen, J., ”More testers - The effect of crowd size and time restriction in software testing” (2013), Information and Software Technology 55, pp. 986- 1003, 2013.

In this study, they found that "[..] a crowd of five time-restricted testers using 10h in total detected 71% more defects than a single non-time-restricted tester using 9.9h". The testers are students and they perform manual testing. Although there are some limitations which could vary to a real test process (to name a few: students, no automation, time-restricted, probably less background information, not end-to-end testing per se), it still shows that more than one person is very helpful in testing.

To summarise: only a developer will (most likely) spot less defects compared to a developer and a tester, even if the total test time is equal.