47

Can anyone provide actual data or scientific studies that show developers are (or are not) less effective at testing their own code than "independent" testers? I've seen many opinions but little hard data.

EDIT: A few followup clarifications:

  1. This is referring to system/e2e tests.
  2. There are also two interpretations of "developers testing their own code": an individual developer testing the code they authored, or more generally developers testing code that may have been written by other developers. Data for either one would be instructive and I think there are different considerations for each. We currently do both.
  3. We use team inspections in most cases for requirements, code, system/e2e tests, etc., with the exception of unit tests. In my opinion this does mitigate at least some of the risk of the test author being "blind" to gaps in what is being tested.

Background:

I'm part of a team of around 20 developers, and until recently we had no dedicated test team. On my first project I was the "software test lead" while still doing actual development, and the expectation was that when you implemented a feature, you didn't merge it until you had also written the tests related to it. We also have a very good culture of celebrating finding (and fixing) bugs, egoless reviews, etc., and to top it all off we have produced very high quality results. In my opinion, it is possible for developers to be just as enthusiastic about breaking their own code as independent testers can be, if you have that kind of culture in place.

I just started reading How Google Tests Software and it seems we aren't the only ones who have had good results when developers are responsible for testing their own code. I haven't even gotten past the introduction yet but the whole idea of having a "Engineering Productivity" group rather than a group labeled "QA" or "Testing", and those people working to enable developers to test faster and better actually sound very similar to my experience: though we didn't have separate titles there are people in our group that are known for writing tools to increase productivity, whether in testing or otherwise, which combined with the culture of quality mentioned above has proven successful.

Our management is pushing for creating a dedicated testing team. They've given a number of reasons, and I've seen many of these reasons/opinions offered in answers to questions like How does a tester's perspective towards software differ from a developer's? and Can developer do automation test for the feature that he has implemented?, but while I've seen lots of opinions (many from experience, and I do respect them as such), I've yet to see any hard evidence or data that developers are inherently less effective at testing their own code than an "independent" party. My biggest concern with moving from our current model to a "separate test team" model is that developers will no longer view quality as their responsibility, and I've seen some of this mentality start to set in on a project where we had a test team. It was an offshore team that proved utterly incapable of writing quality tests, but because we didn't ditch them until a year into the project, we built up a lot of quality debt while also allowing many devs to get out of the mindset of also being testers.

I'd like to push back on this idea of separating development and testing but want to do my homework first. There are opinions and anecdotal/experiential evidence that support both viewpoints, but I'm interested in knowing if there's any actual data to support one or the other (or "it depends").

  • 4
    The developer cannot do all tests and be the only tester BUT s/he has to test first its own work and then share it with others. In many companies the "testing" of developer is limited to successful compilation without errors - and then the developer think s/he has done his/her job. The final quality and speed of development of such companies is bad. – i486 Aug 4 '17 at 19:25
  • 2
    +1. Excellent question. Questions like yours makes it worthwhile to skim over daily dose of "please fix my XPath" questions (or worse: "here is my homework, do it for me ASAP or else"). – Peter M. Aug 4 '17 at 20:34
  • 1
    I have actually decided to remove my answer. The whole point of the answer was based on me missing a key point of the question - you don't have quality issues. Even though my opinion still stands, it does not apply to your case, so it's even more off topic than the other answers you have. Still, I think your question is missing something crucial: why does management want to create the testing team? The reasons you've provided are basically fixing what is not broken (its why I have missed the key point). I think you are looking for data, and you have the data at hand: your own team experience. – Krystian Aug 5 '17 at 14:38
  • @Krystian I agree, and I've tried to push on that very point--"what problem are we solving". I hesitate to try listing their reasons right now since it's been long enough since the last time I asked that I don't fully remember their reasons, but I think it was mostly "we should have a test team" and a desire to speed up the development process by dividing the work. Plus various opinions that I alluded to about a separate team being "better" though our results have arguably shown it to be unnecessary. – c32hedge Aug 5 '17 at 14:54
  • @c32hedge it is possible, that the QA team is required by your clients, and you just have to comply with some standards they expect. – Krystian Aug 5 '17 at 17:19
31

There doesn't seem to be a lot of research data on this; this is what I found:

Waiting for builds:

This article about "Why software testers can't test" quotes a survey from IBM (The Future of Testing: Where Do Testers Spend Their Time? 2014):

59% of those polled, the largest percentage for all the available choices stated that the one activity they wish they could spend less time doing was…”waiting for test assets.” Dagger.

This has also been my experience as a manual tester in the past. It even gets worse as you find some critical defects that block your testing. You need to hand it back to the development team and wait for the next testable version.

Shortcuts:

I have read an article a while back (but can't find it anymore) that stated that as their company switched to a dedicated QA-team the overall quality of their product started to drop. As the developers thought the testers would test it thoroughly, the developers stopped most of their testing efforts. Their work got only a little bit worse because they didn't find obvious issues instantly. The testers were under more time pressure than the developers. They were lagging behind, while the company wanted to ship. The testers started taking larger and larger shortcuts, missing more and more important issues.

On a personal note, fixing issues found by a separate team are harder to fix due to context-switching and forgetting how that feature was built. This should be provable with data: there is a lot of research about multitasking and context-switching.

Companies are cutting their QA-teams:

Dedicated QA-team:

I am all for a QA-group, as long the members are part of dedicated software development teams. Where they help with the whole team approach to quality and quality-build-in. Preferably helping the team to do test-first practices.

But having a dedicated test team that you hand over a testable product to... that feels like a horror I hope to never again experience in my life.

I love swarming. Swarming the requirements, swarming the architecture & design, swarming to create automated tests and code, swarming to refactor and then swarm exploratory testing sessions as well.

Training:

You can train developers to switch between build-thinking and critical-thinking. Having critique on the product is just another mindset. Most developers are smart enough to be able to master these skills. You just have to motivate them that it is important. I truly believe a very good developer must also be a very good tester.

  • 5
    +1 for: as their company switched to a dedicated QA-team the overall quality of their product started to drop - makes sense, because devs started thinking that quality is someone's else's responsibility, and can be tested out later. Quality has to be build in. Would great to have the link. – Peter M. Aug 4 '17 at 18:37
  • 4
    @PeterMasiar Had the complete opposite experience: Devs testing themselves are bored and annoyed by testing tasks -> less productive in testing AND coding. The trick with a dedicated test department however is to couple it closely with the dev team - at least the head should be in the same room/floor and closely involved with at least the development head. Sure, devs should write unit tests and make at least happy path tests, so no time is wasted with totally untestable features. But for extensive UI based tests, I love having actual testers! – Frank Hopkins Aug 4 '17 at 22:34
  • 2
    @Darkwing - I mostly agree with your "complete opposite" experience: In my ideal world, devs would pair with QA automation testers to write e2e basic tests (and add locators as necessary). When all semantically significant web elements on a page have a good locators, QA testers can continue developing more complicated tests without devs. But if QA testers have no or little support from devs, they have to rely of clunky or flaky locators, like XPath we are getting in questions all day every day here. To me, lack of stable locators is a sign of lack of cooperation between devs and QA. – Peter M. Aug 4 '17 at 22:44
  • After some time, the test number gets so high, and setup so complicated, that QA is necessary just for maintenance. – charlie_pl Aug 5 '17 at 9:19
  • 1
    I cannot imagine a company without a dedicated QA team. After big changes, e.g. refactorings or big new features, you have to retest the whole app. Also, QA are not only testers, they should control quality even before the development starts - they should be active in the specification/analysis process, too. – Sulthan Aug 7 '17 at 10:53
25

Disclaimer: this answer is purely based on personal, anecdotal experience.

In my 10 years as a Software Developer, I have known 4 different situations:

  • no tester (besides myself),
  • tester in another team, large delay (weeks),
  • tester in another team, short delay (hours),
  • tester in same team, short delay (hours/days).

Note that Unit-Tests were always on my end, the tester instead would be in charge of high-level tests, either manual or automated.

Spoiler: the subjective evaluation of the quality of the delivery has informed the order in which I listed the situations.


Tester

First of all, in my experience having a second pair of eyes is invaluable:

  • a different reading of the specifications/requirements help identify ambiguities (arising as the tester expects a different output than the developer),
  • a different person simply thinks about different tests to do.

In short, as a developer, if I thought about a corner case, my code will handle and my tests will show case it. The issue, however, is about the corner cases I didn't think about.

Secondly, a professional tester will develop instincts of things that can go wrong and my (sole) data-point has been that they were much more vicious in the situations they came up with (a lot more variations than I had thought of).

Therefore, a professional tester is invaluable when it comes to raising the quality of a project.


Delay

Maybe simplistic, but my best experiences were when the tester would test quickly.

Context-switching is generally branded as lowering productivity, however it has been my experience that is also lowers quality.

The better recollection I have of the data-flow, specific concerns, etc... the more likely I am, when fixing a bug, to:

  • avoid introducing another bug instead,
  • think about potential other buggy situations and fix those too.

This means that a shorter delay between production and feedback not only increases productivity, but also quality.


Them vs Us

I have not had a problem with individuals, but I have regularly butted my head against different priorities.

When the Dev Lead and QA Lead disagree on the priorities, and this means nobody can test my delivery quickly, then... it introduces delay, and all its consequences.

It also engenders some degree of frustration; or even led to functionality being delivered in production before being reviewed by QA. I guess it could help keep developers on their toes... but I do not recommend it.


Involve

Overall, my best experience has been with a dedicated tester in the same team, and office.

By involving the tester from the beginning:

  • the software is designed to facilitate testing,
  • if necessary, testing-only requirements are incorporated in the design,
  • being involved in the design, the tester knows the intended functionality and requirements (beyond the written specifications),
  • being situated close-by, the tester and developer can ask for clarifications on the fly.

The latter part has been instrumental in raising the quality of the product I was working on.

If you can speak to the tester/analyst to ask for a clarification, you are likely to do so. If you can call them, you may very well do so. If it requires an e-mail... you might do so... but probably will not. Because you are writing the code now, and you are not going to pause until they read your e-mail and reply. This is the difference between clarifying something, or assuming "for now"... and we all know how that ends up.

The same goes the other way around. Rather than waiting on the implementation of the test because it's unclear how to use your code, it's much simpler for you to answer now. And the faster your reply, the less likely they are to put your testing on the back burner as they switch to another task.

Of course, interruptions can be annoying, etc... however, a once in a day clarification from one person is not the end of the world whereas a once in a day bug introduced is a significantly lower quality.


Conclusion

In the end, my experience is that the best quality results from having a dedicated knowledgeable tester:

  • involved from start to finish,
  • in close proximity,
  • providing feedback in short order.

Having them in the team is not necessary, but helps on all points, and probably helps switching the "them vs us" mindset off.

Note: it does not necessarily mean having the same person to report to; my company at the time used project-teams which assembled people from different actual teams and put them together in the same office.

  • I agree with your sentiments in general, but I think you are mixing approaches beneficial for unit testing, and very different approaches beneficial for system/e2e testing, which could lead to confusion. I believe that OP is concerned about unit testing, I asked OP to clarify. – Peter M. Aug 5 '17 at 19:21
  • Also, calling a professional tester vicious reveals you being locked in a developer mindset. I think better name for the approach is adversarial mindset. Developer tends to think about how to satisfy "happy path", while (good) tester tries all invalid inputs to check proper error handling, before app is released to the open world. Such behavior might seem "vicious" only to a developer whose ego is invested in ownership of the code, not it validity of the code. – Peter M. Aug 5 '17 at 19:28
  • 2
    @PeterMasiar: I think that if it was a matter of ego, I would not be praising the viciousness of the testers I worked with. I do think you are right about the "happy path"; it's generally hard enough to make the application work correctly (and swiftly) in the correct cases that it mobilizes ones' thoughts, to the point of casting shadows on edge cases and errors that have to be handled. I greatly admire people who are rigorous and creative enough to uncover those forgotten cases. And I'll still call them vicious, or devious, depending on my mood, with a huge grin plastered on my face. – Matthieu M. Aug 6 '17 at 10:19
12

TL;DR: Devs write own unit tests. If you want different eyeballs to look at the unit tests code, code review is the answer.

EDIT: OP clarified question: it is about e2e tests. For e2e tests, using separate team of developers is possible but adds different kinds of challenges. Added to the bottom.

Related question: How does a tester's perspective towards software differ from a developer's?

Quality cannot be bought, but has to be paid for - no shortcuts


No studies, just common sense.

Developers should write their own unit tests (and fix the broken ones), because most recent developer knows the best the recent changes, and is in the best position to investigate the consequences.

It makes no sense to use different QA test team to write and fix unit tests, because they will have to learn what developer just learned and did (and devs have in operational memory), ask them many questions why (and possibly create misunderstandings). Just huge duplication of efforts, total waste of time. All those questions should be asked during the code review and resolved by original developer.

Separate QA testers (if used, a big if) are best positioned to write system/integration/e2e tests, because those tests may (or may not) benefit most from different set of eyes, possibly using different unsaid assumptions, and sometimes using different programming language: core devs might be more concerned about performance, while QA e2e testers are much more concerned about the productivity (if testing needs to run faster for valid business reasons, buy more servers and run test in parallel).

But often, devs are also writing system/e2e test, so they can fix any changes in the UI, because they are best familiar with the changes (and less surprising: original developer does not need to investigate what changed).


E2E tests are very different animal. With those, second pair of eyeballs might be beneficial.

Again, no studies, just real-life experience earned the hardest way: by own sweat and blood :-)

In our company, we use all three approaches:

  • developers also writing e2e tests (maintaining tests written both FitNesse, old homegrown automation framework, and new test using new homegrown framework enhancing Selenium WebDriver)
  • separate test automation coder (just one :-( ) catches up with e2e tests which were not developed during original development (business decided to take shortcuts)
  • tests are developed using hybrid approach and pair programming.

Automation tester (me) developed in-house framework for Page Object testing, and helper functions using it, simplifying the code and increasing productivity. But e2e tests rely on convenient reliable locators (IDs and names) for all semantically significant page elements. Which is the root of the problem.

As you can imagine, developers do not like learning additional framework, even if it improves productivity when writing e2e tests.

OTOH, when backfilling tests, sometimes convenient locators are missing. Workaround is to use flaky XPath locators (with all the involved problems - better avoided), spend some more time to develop CSS locators, or ask developers to add convenient locators (names/IDs) to existing code.

Adding locators is easy, but scheduling it is hard. It may take hours or days (depending on developers workload, willingness to cooperate, and ability to switch context), and test have to wait for those new locators (and forces them to switch context while waiting for improved locators), so adding locators on demand approach is far from optimal.

Best approach (in our experience) is pair programming for e2e test development (pairing automation tester with developer) to develop page object for new pages and smoke test touching many/most widgets on the page. Automation tester knows the automation part and how to do it, developer knows how to add locator when needed. Using this approach, adding locator and using it in a test takes seconds, not hours or days.

Afterwards, automation tester can work separately without core developer to add more and more complex tests, reusing the locators.

But without this "paired" step, developing locators is exercise in frustration for automation tester:

  • it either takes very long time waiting and swapping context,
  • spends lots of time designing location strategies using CSS and other attributes which are available,
  • or uses flaky XPath.

We are transitioning/settling/improving our process for pages using e2e test with Selenium WebDriver. Single-page application pages with Angular make it more complicated.

Angular and single-page app design pattern also opens yet another can of worms: who is responsible to fix e2e test when page fails after UI design change.

My feeling is that test should be split into (at least) two groups:

  • smoke test, developed by pair programming as mentioned above. Test failures (changed locators) are fixed mostly by UI developers themselves.
  • more complex tests, developed and maintained by QA test automation team.

Of course this requires very tight cooperation between developers and QA test automation.

In a "throw it over the wall" approach (pages are dumped on QA automation without decent locators and without the ability to add necessary locators quickly and conveniently), productivity of QA automation will be very low, tests will be flaky and unreliable, and pushback against e2e testing as waste of resources will be strong.

Add to it the temptation to use automation testers with lower-level skills, and disaster is inevitable, as we can see from questions asked on this forum daily, which are trivially answerable by better locators and trivial amount of programming skills.

  • 2
    Grrrr, those are all my thoughts exactly. you thief you ;) +1 of course – Michael Durrant Aug 4 '17 at 18:16
  • 1
    Exceptions aside, unit tests should check public methods. And the class should be documented clearly enough that any developer can easily write a test for it. If you need the devs who wrote the class to write the tests, there is something wrong with your design. (Not saying that they shouldn't, I would as well consider it a devs task to write unit tests for classes he writes, but knowledge of the internal workings should not be a factor!) – Frank Hopkins Aug 5 '17 at 1:37
  • I am not saying that nobody else is capable of writing unit test - do you see any strawman?. What I am saying that person who changed the code is the person who has most current knowledge about the code, so can write test for it in least amount of time (without spending additional time to learn it). It is all about being productive, delivering more working code in less time. TDD says that (failing) test should be written before fixing any bug in code, so in TDD approach, test and code changes are obviously written by the same person (or same pair, if using pair programming). – Peter M. Aug 5 '17 at 18:51
9

Disclaimer: not hard data, but something from a personal experience.

We currently have a good representation of why developers should write unit-tests for the changes they made (at least, critical and most error-prone parts if under time-pressure, as we are).

One of our products is an AngularJS application, the front-end of which is being developed by several UI JavaScript developers. Our JS side is tens of thousands of lines of JavaScript without a single unit-test. "We don't have time for it" is one of the explanations UI developers might provide when asked why. In other words, our dev side is not participating in any testing activities.

Our QA team is trying to compensate for the lack of unit tests by adding more end-to-end browser automated tests (we currently have about a thousand of them), which basically flips the famous pyramid upside-down.

This has multiple consequences:

  • we often have a manual testing period when we (QA team) find trivial issues which could be quite easily caught with unit-tests
  • sometimes this kind of issues make it further to staging and production which makes them more expensive to fix
  • end to end tests are really slow and flaky
  • it is often required to research an end-to-end test failure to find the actual cause of it - the feedback loop is really slow
  • we also have some problems communicating markup changes
  • 2
    +1 and ug. someone in app dev really needs to drive using TDD and pick a date to do it for all new or newly changed code going forward. Personally I don't consider that unit tests have to be written before app code but it is preferable. The only good (not using 'rule' or 'best' anymore) guideline I follow is that both will exist when the code branch is submitted for review, test and merge. – Michael Durrant Aug 4 '17 at 18:24
  • 1
    compensate for the lack of unit tests by adding more end-to-end browser automated tests - you are doing it wrong, and you know it. It does not make any sense to add e2e tests until you have comprehensive suite of unit tests first: if unit tests fail, it points to error quite closely. When e2e test fails, investigation starts. – Peter M. Aug 4 '17 at 18:30
  • 3
    @PeterMasiar Indeed, I think the answers is pointing out that this is caused by the separate QA team. Which also leads to the developers thinking they do not need todo extra work for quality, making things worse. It just the common "separate test team" pitfall perfectly described. I don't think it wasn't meant as a solution :) – Niels van Reijmersdal Aug 4 '17 at 18:35
  • 2
    I was speaking from my heart in hope someone can share my pain :) Thank you, yes, we are definitely doing it wrong. – alecxe Aug 4 '17 at 18:36
4

From experience, I have found that developers are very effective at testing what the code does, but not as good at testing what the code is supposed to do.

The first example that popped into my mind is a GUI I designed. I tested the daylights out of it, and though it was flawless. Brought it in front of my customer, and 10 seconds in he had crashed the system. He was kind about it, but afterwards, I had to figure out why a system I had tested failed so miserably.

The answer was that in my testing, I'd click buttons. I'd do

  • Click button 1 - make sure it does X
  • Click button 2 - make sure it does Y
  • Skip button 3 - I know it doesn't work yet
  • Click button 4 - make sure it does Z

How the heck did I miss button 3, when I knew it was broken? Well, I did my testing so many times during the build that I developed a habit. I knew button 3 didn't need to work yet, so I kept skipping it. Then, when it needed to work... I kept skipping it.

There's plenty of approaches I could have done to resolve this (automated testing from the start would be one), but the anecdote highlights a key challenge for developer testing: we know what the code does. In our head is a giant mental model of everything that block of code is doing, and we subconsciously leverage that model during testing.

I think you can teach developers to do good testing, but I think you have to actively do so. It's hard to step back from your own model and approach software from a blank slate. That's why we have code reviews.

  • +1 ... and that's why you need another human mind reading the requirements and checking if code fulfill them and can handle surprises like invalid input. And some of these test can be automated too. – Peter M. Aug 7 '17 at 19:56
2

When working on something complex, there is always the risk of bias, blind spots, tunnel vision, and stress induced malfunction of the thinking parts of the brain. ( one good book, though from outside the IT world, is Processing Under Pressure, by Matthew J Sharps)

That is why it's always good to have another human around,
not a yes-person or naysayer, to have another look and guard your back. For code, but also for buying new paint for your house, buying a new suit or writing an important letter.

This requires an environment where this friend with the title * honorable enemy * can function without getting beheaded. Many multimilion dollar movies flopped because the director only accepted yes-persons to review the ideas.

It is possible let a colleague developer do the testing. Except a similar background and similar way of thinking can still lead to the same blindspots. Enters the tester, or test role.

Dangers: silo thinking, us vs them, throwing over the wall attitude. If developers see testers as 'failed developers', testers see developers as 'reckless cowboys', end users see both as 'geeks from Mars', end users get seen as 'those (dumb) end users', ... then nothing can save the product.

Testers as part of a team, with short communication lines, where everyone is interested and dedicated into creating quality, it can be done.

  • Thanks for the edit c32hedge, one phrase less and the text has become better. It does prove the point that having a second reader (or tester) is needed, because at the time of writing I was convinced that the text was "perfect" then the edit showed the glaring obvious flaw. Blind spots while busy creating text, or code, or configurations. – W. Lin Aug 9 '17 at 6:48
1

A few years ago I've done a quite extensive literature study into testing, especially regarding benefits of testing. I have not encountered a scientific study in precisely what you are asking (developer versus tester), but I did encounter a quite interesting scientific study: Mäntylä, M. V. & Itkonen, J., ”More testers - The effect of crowd size and time restriction in software testing” (2013), Information and Software Technology 55, pp. 986- 1003, 2013.

In this study, they found that "[..] a crowd of five time-restricted testers using 10h in total detected 71% more defects than a single non-time-restricted tester using 9.9h". The testers are students and they perform manual testing. Although there are some limitations which could vary to a real test process (to name a few: students, no automation, time-restricted, probably less background information, not end-to-end testing per se), it still shows that more than one person is very helpful in testing.

To summarise: only a developer will (most likely) spot less defects compared to a developer and a tester, even if the total test time is equal.

  • 1
    So I'm guessing this meant five people for 2 hours each, versus one person for 9.9 hour? That makes sense intuitively because the longer you test something, likely the more bored and undirected your testing will become. Whereas with multiple people, you're multiplying that initial "burst" of testing energy and likely applying it in different places. Of course, if what you're testing is fairly small, you may need fewer "bursts" to test it "enough". – c32hedge Nov 13 '17 at 14:09
  • @c32hedge: you are exactly right. I guess you could say that developers tend to be more bored and undirected if they do their own testing at the end of the process. If another person (a tester) tests it, it is at the start of their process, which will have the "burst". – Renzeee Nov 13 '17 at 14:31
  • Might depend on the developer, but yes it's a good perspective :) – c32hedge Nov 13 '17 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.