5

We're building two inhouse products:

1- The first has web, android and iOS (total of 9 developers) versions and has 2 QA engineers (1 full time manual/automation engineer, the other is a 50% dedicated manual QA engineer).

2- The other product has a web version (3 developers) with 1 full time manual/automation QA engineer.

Our QA team for project #1 is clearly outsized by the development team productivity of releases. This is delaying getting valuable feedback on our releases.

Although we're trying as hard as possible to hire new QA resources, this is proving challenging and time consuming. We have deadlines looming by (product #1 in 1 month, product #2 in 2 months) and we need fast and effective ways to handle this shortage in QA.

Are there any tips you can provide for such scenarios.

Thank you

  • Not sure what you mean by "QA team for project #1 is clearly outsized"? Your title says "understaffed". Do you mean "undersized"? – Peter M. Apr 4 '18 at 14:28
  • What's your automation strategy? – Vishal Aggarwal Apr 4 '18 at 21:30
  • Basically how and what you cover in automation? – Vishal Aggarwal Apr 4 '18 at 21:31
  • @PeterMasiar I think "understaffed" and "undersized" in y context serve the same meaning. – SaryA Apr 5 '18 at 6:59
  • @VishalAggarwal unfortunately, we've started recently building our automation scripts and we're still not using it to its full power now. Is this bad? Yes. Should we have done it from the beginning? Yes. – SaryA Apr 5 '18 at 7:00
6

I don't think your team is ridiculously understaffed. We run our operation more lean than you do.

In real life, ratio of developers to integration/system/functional automation testers is as low as 10:1.

How is possible to maintain quality? By focusing more on unit tests (written and maintained by developers).

Unit tests should be your first line of defense. Because if unit test breaks, it is pretty obvious what broke and how to fix it. Many projects have more lines of code in unit tests than in the production code. And use coverage analysis to make sure that unit test exercise most of the core code.

With such comprehensive battery of unit tests, automated UI/functional/system level test need to cover only 20% of code which covers 80% of common functionality (to keep reasonable quality with least amount of automation). Automated regression tests are code, and needs to be maintained when tested code changes, so you want to have least amount of the regression tests (so you have less test to maintain).

It seems to me that problem with your projects is not the low number of QA testers, but insufficient unit testing and low unit test coverage.

See also debate about test pyramid and Pareto principle about the need to focus.

  • " 10:1" ratio - are you sure of this? It seems a bit overstretched. "Unit tests should be your first line of defense." - I 100% agree. Unfortunately, our unit tests are outdated. One of our devs is trying to free up some time to maintain them. And yes, writing/ updating unit tests should be a continuous things not something we do in spikes. "use coverage analysis" - that's a very good suggestion. " [automated] tests need to cover only 20% of code which covers 80% of common functionality" - any rules how to decide what needs to be covered with these tests? – SaryA Apr 5 '18 at 6:56
  • 1
    10:1 is very lean but possible with excellent other tests. Management needs to be aware that it is rather lean and bugs might slip to production, and willing to take the risk to save budget. 80% of the common functionality: easiest and cheapest is to use expert opinion :-) – Peter M. Apr 5 '18 at 13:51
  • Well said, @PeterMasiar. OP might be at risk of falling into the trap of QA being 100% responsible for quality, which is a path no team should be on. youtu.be/yRP29wFqu20 is a great presentation on "Quality at Speed: How Atlassian does QA" – Subjective Reality Apr 8 '18 at 2:18
  • Thanks Peter for your valuable input, and @SubjectiveReality for the share. – SaryA Apr 9 '18 at 8:03
  • @PeterMasiar : "In real life, ratio of developers to integration/system/functional automation testers is as low as 10:1." Your real life is much different from my real life. Could you provide some statistical resources because it sounds surprisingly to me and really worth researching. – Alexey R. Apr 9 '18 at 11:33
3

Risk based testing can help you with quick feedback

This is another good aspect that you should look into.

As stated in one of the answers, developers should be asked to test their work. This is definitely going to help. I completely agree with that.

But, in addition to that "Risk Based" testing can help you further.

  • In risk based testing, you try to ensure that most common application flows are tested.
  • This way, you might miss some of the "rarest" scenarios but at least you will have a list of most common workflows that are not working.
  • So, you pick "High" and "Medium" priority test cases and test them. You leave low priority cases. If the time crunch is too much then you might end up testing “High" priority test cases only; for the purpose of initial feedback.
  • But, that doesn't mean that the testing is complete. But, that does mean that commonly used application areas are either "working fine" or "have issues".

Once you have recruited QA resources and your team is NOT out of balance anymore, you can cover what was missed earlier because of lack of time.

But, this gives you a fair idea (if not complete) of project health. And developers will get important defects to work on.

  • That's a good approach. In retrospective, it sounds common sense, but I didn't clearly think this way until you proposed it. Thanks – SaryA Apr 4 '18 at 6:46
1

As it seems to me, the ration of dev-to-test in project#1 is obviously overoptimistic. One dev might change one line of code that would require huge test efforts.

However once you faced such the situation (looks like there were some missing risk assessment on project start if you have such the ratio) I would recommend to force your devs to test (give them a chance to demonstrate their productivity one more time) unless you have enough QA resources hired. After all this is what Scrum declares as a value - team cross-functionality.

  • Thank you. My understanding of "a cross functional team" is a team that has members that cover all needed functions for it to operate, not to have each member being capable of doing different functions (although the latter is good whenever feasible). Our developers are under a lot of (hopefully healthy) pressure, and although they should do their own developer tests, a solid layer of official QA is needed (detailed test cases execution & regression testing). – SaryA Apr 3 '18 at 21:58
  • 1
    I believe there there is no magic pill that would let your 1.5 QA guys test what your 9 dev guys produce. You need to restructure the tasks. Remember the "iron triangle of project management". – Alexey R. Apr 3 '18 at 22:11
  • Restructuring tasks (prioritizing urgency of QA tasks) is one way we could handle it. Thanks – SaryA Apr 5 '18 at 6:58
0

The team size doesn't seem to be the main problem here. If you are currently recruiting with deadlines approaching within 1-2 months then this was a management issue. This should have been brought up earlier so you have the time to employ more staff, as it will still take any new tester a few months to get use to a new system to be of any use, but I digress.

Since you are already in this position what I would do is try and shift the mindset. Although its easy to throw testing at testers, it should be the responsibility of everyone to ensure quality software.

  • If you have a deadline in 1 month, does this mean all the necessary features have been implemented and the code is now, or close to, being froze? At this point - as others have suggested - the developers can now help with testing.
  • If you are still adding new features and are using agile, this is where you start dropping less important work to ensure a decent minimum viable product can be released that still satisfies the customers requirements.

Adding a new tester now will likely not help with the short term goals as you will have to spend time to hire, teach them the new system, etc.

So, my advice would be to try and make sure everyone is focused on quality. Try and get developers, managers, anyone within the company who can use the software to use it, focusing on as much coverage as possible. Good luck.

As a side note: If you have time, I suggest How Google Tests Software as it has some interesting insights into their strategy for testing very quickly and, more importantly, ensuring a quality product is delivered without having thousands of testers.

0

I quite agree with PeterMasiar and Aalok. Good unit testing can nip some code issues in the bud and priority-based approach may help reduce testing to a reasonable optimum. There are also some measures a tester can take to optimize regression testing, a largely disliked but necessary type of testing. I believe in your case there are 2 viable options:

  1. Two-level approach to regression testing. You just divide regression testing into iteration regression and full regression. Iteration regression concentrates on features and changes made in the iteration and areas of the application that could be affected.Full regression covers the whole application and is usually performed before major releases.
  2. Risk-based approach Aalok offered can also be extended to regression testing.Testers concentrate on high and medium priority regression test cases. This way they fully cover critical system functionality and exceptional conditions (negative test cases, boundary value test cases, etc.).

Going back to overall testing, exploratory testing may also help detect bugs saving time. But here a tester only relies on his or her experience and 'gut feeling'.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.