10

We've been growing as a company and with it the development, QA, DevOps and other teams.

I could not help but notice that with each new addition to our QA team, our meetings are becoming longer and, in some sense, less productive since some of the topics are becoming off-topic for other people; we are handling more products and processes than before and it is becoming challenging to keep testers "generic" with a broad knowledge of the whole ecosystem and the products under test.

This is probably a "good problem to have" in a sense, but are there general recommendations of how big can a single QA team be? What factors and metrics can help with that?

Improve this question
5
  • 3
    You might have heard about The Mythical Man-Month. Although it doesn't concern QA directly, it does concern team size in a software development context.
    – codeaviator
    Commented Dec 16, 2017 at 1:54
  • 2
    This debate has always seemed a red-herring to me. More important is to ensure developers do at least some basic verification, documentation and regression-testing/ unit-testing/ scoreboarding (optionally with SETs/testers embedded in the team). And there should be some SCM process, doesn't need to be CI, but need to prevent crap getting into branches, and detect early if it is. But having one big monolithic QA dept firewalled from the development side of the house which throws barely-working code over to it constantly, then arguing about the optimal size of that dept, is a total red-herring.
    – smci
    Commented Dec 16, 2017 at 11:03
  • 3
    The more relevant question is: are testing responsibilities spread, not just dumped on hapless QA? How functional is the working relationship between dev, DevOps and QA? When crap gets checked into dev branches, is it detected by test fails/ code reviews/ regression fails, or does it wait months before QA finds it and by then its too late because it was bundled with functional changes/ enhances? Basically, QA is not a wish-fulfillment object for a company-wide lack of structured development+test process. Although it's often treated like that.
    – smci
    Commented Dec 16, 2017 at 11:06
  • 1
    So the question's entirely the wrong question to ask. Also "What factors and metrics [for QA]?" is another wrong question to ask, it leads us to bodycount-style dept-level nonsense metrics like active bugcount, time-to-close, which don't give any higher information. Not even how quickly holes in test coverage got patched. Let alone organization-wide metrics like "Can we define, spec and ship a body of code within n months that actually roughly meets customer needs?" (agile vs waterfall is another wrong debate. Just make the code work. Whatever process works for that customer segment).
    – smci
    Commented Dec 16, 2017 at 11:13
  • 2
    @codeaviator - I wonder how many questions would be answered if people just read this one book. After 35 years, still holds like no other IT book.
    – Peter M. - stands for Monica
    Commented Dec 19, 2017 at 15:41

9 Answers 9

Reset to default
10

You're going to get all kinds of recommendations on QA team sizes, from zero to twice the number of developers, or higher. The right number for you depends on what you need to test, what kind of testing your developers do, and how solid your software needs to be.

My experience is that small teams -- no more than a handful of people -- are more effective than big ones. As you've seen, bigger teams need disproportionately more time on communication and coordination.

You may have reached a point where it's unrealistic to expect every tester to be generic with broad knowledge of the whole ecosystem and the products under test. Maybe that's ok. Consider splitting your big team into smaller teams who specialize. Try to put at least one generalist on each team.

Improve this answer
7
  • 2
    twice as many as devs? Do they have a theory behind it?
    – Yu Zhang
    Commented Dec 15, 2017 at 19:12
  • 2
    @YuZhang Devs create twice as many bugs in a day as a human can find? Sounds right, based on my own code.
    – anon
    Commented Dec 15, 2017 at 23:00
  • @YuZhang Maybe when reliability is worth the expense? Nuclear plants, space launches, that spreadsheet for Carol in Accounting, etc.
    – corsiKa ♦
    Commented Dec 16, 2017 at 2:17
  • 1
    @YuZhang - I've worked in organizations where the ratio of dev:tester resource fluctuated between 1:1.5 to 1:1.7 - and we felt like there weren't enough testers some times. I really wouldn't be surprised if I came across an org. with 1:2.
    – shalomb
    Commented Dec 16, 2017 at 2:21
  • 1
    @QPaysTaxes, thanks, the highest QA / Dev ratio suggested to me was 1:1, it has never occurred to me it would go over 1:1.
    – Yu Zhang
    Commented Dec 16, 2017 at 3:47
4

You might want to consider breaking your QA team into smaller groups so they can focus on specific products. Keeping QA engineers 'generic' doesn't really work for agile teams, especially when the application engineers they work with are allowed to specialize.

Improve this answer
4

TL;DR: The smaller the better, I would aim for 3-6 people.

The overhead of communication clearly grows when your team size becomes bigger. I guess the tipping point is around 10-12 people, now to many influence each other and your in for some real communication pains.

I like to keep a a team to a maximum of 5-6 people doing the actual work and only handful of stakeholders (3-5) pouring in work in the teams backlog. Then the team can decide what to pull in. If there is more work than a single team can handle I would vote for a second team on the same backlog. Now the teams have to figure out some coordination which does not include the full teams to often. The simplest form could be a daily Scrum of Scrums. Don't think it really matters if this is a QA-team, a Dev-team or a Sale-team. They could function and organise in the same way.

Larger teams also have higher turn-arounds, where people come and leave (team or company). Bringing the group development stage back to forming. This is very ineffective. Keeping teams together is better and much easier with smaller teams.

The smaller the team and the more self-organised the better, but there is also a minimum of around 3-4 people. Try limiting the bus-factor to around 3, this means that team-members should not act as individuals.

Improve this answer
1
  • 1
    +1, similar to "surgical team" as defined in The Mythical Man-Month. It is amazing how well the book holds: after 35 years, still mostly valid. No other IT book is so relevant after so much time passed.
    – Peter M. - stands for Monica
    Commented Dec 19, 2017 at 15:39
2

I don't think there is an "ideal" size for a QA team - it may depend on project, company, and many other factors.

Generally, the size of the QA team should not be big enough to be a bottleneck, but if the testing becomes bottleneck, then you'd increase the size of the QA team.

It may be pretty difficult to give an ideal number or even hard numbers given the fact that testing is a very skill dependent. It may happen that your developer have a good amount of output and there may be 2 or even 3 QA to test that (yes I have seen that and have been one of those QA. Or vice versa, where 3-5 dev works are easily being managed by a single QA.

Also increasing the head count or having a fixed ratio of devs:QA is a bad idea - as mentioned here http://kaner.com/pdfs/pnsqc_ratio_of_testers.pdf

The level of complexity of a project may also decide how many QA's you or your team might want to have in order to have a good quality project delivered, and have the complexity covered. Quality bar of a project is a key factor in deciding QA's in a team. A client-delivery project might need or have more QA's than an internal leave management system project.

Improve this answer
1

At my past work the QA team's size was about 30-50 people. However, the QAs were distributed to development teams. So we averaged about 2 QAs per 3-4 developers. On average the ratio was 1 QA per 2 developers.

The QAs were mostly manual (~99%). They did prove to be a sufficient bottleneck (although they hated being called that). We had a separate "automation team" that provided automated testing for the "entire company" based on the most critical features of the product.

So, to answer your question: I'd start by identifying how much of a bottleneck is QA (if at all)? That should inform you about the QA size more effectively (irrespective of whether they are manual or automated).

For example, since QA was manual on our teams, the dev capacity of work was 2x that of QA. However, the QA work in theory was almost 1:1 with respect to "amount of time/effort to do the job". It's possible we could've done better with 1:1 dev:QA ratio, but budget wouldn't allow it. The communication overhead too would've been too much. However, we never tested this hypothesis. I'm not sure if there are any studies on this, but empirically this is what we found.

I'd strongly suggest finding out the QA pace of work - how long does work take to go into the QA team and out of it. Using that info and comparing against feature delivery efficiency, you could have a good judgment on the potential size of the QA team. You may have to experiment if you don't have the budget - e.g., get some devs to be QAs for a bit or have the QA managers to chime in to increase the resource pool, hire contractors etc., or just get creative!

Improve this answer
1

I manage a QA team and ask this question to every interview candidate. I mostly get answers that range between a 1:2 and 2:1 QA to Developer ratio.

Ultimately, I think the advice from @user246 is on the right track...it depends on how complicated the test subject is, what kind of testing your developers do, and how solid your software needs to be.

Find some good metrics to help you figure out what the right balance is. I watch the number of new issues created for developers and how many issues get tested every week. If I see a trend of developers creating more work than testers can keep up with, I start asking for more testers and I have some solid data to back up my request.

Improve this answer
1

The most effective and productive teams I have seen were known as feature teams having only 3 members- 1 business Analyst , 1 developer and 1 tester where all 3 are more of generalists than specialists.

At first glance, it may look like not effective composition but in the long run it is effective and very efficient where they truly work as a team to finish an feature in short sprints by having very open & transparent communication where feedback loops are very short sometimes invisible.

Improve this answer
1

For team of N people, to communicate between each member you need N*(N-1)/2 links. It is a quadratic function, O(N**2), so obviously does not scale well. To avoid low Bus factor, which tries to skew the number in opposite direction, best size is 2.5: one subject matter expert, one pairing/learning to become one, and one having overall idea and being able to step in if necessary. And have multiple teams, one for each specialized competency area.

Of course this requires specialization, some people think that Specialization Is For Insects and you need competent generalist QA - but can you afford them, and pay for the time required for the cross-training?

Improve this answer
1

It is time to specialize and create smaller expert teams. Possible scenarios how to do it:

  1. domain knowledge teams
  2. tool knowledge teams
  3. testing types knowledge (black-box team & white-box team)
  4. platform-based team

I would probably start with dividing people into test (people who test the software) and QA (people who prevent defects to appear in a software) teams.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.