Mobile automation: Boss wants 100% coverage. How feasible is that?

Question

Just started a new mobile automation role using react native for iOS.

I'm not new to automation, but I am new to React Native (with detox and jest). The learning curve is slow, but we are getting there.

He wants a complete regression pack. That's fine...no problem.

However, he wants 100% coverage. That's my issue.

I will be using grey box testing. However, I do think certain principles are universal regardless...

I was always taught 'don't automate all the things....automate the RIGHT things'.

I will automating with a simulator (i.e. iPhone X). The thought has occurred to me. Can I automate with real device in grey box testing? As I would like to as then I will get more robust smoke and sanity test.

As I'm new to this role and still learning, it's discerning the RIGHT things the because I don't want to create unnecessary tests that create road blocks.

As I'm starting this framework from scratch I am wondering, if anyone has been in this position before?

How did they deal with it?

Answer 1

Your boss doesn't want a flat "No" or to hear their request is impractical.

They want to reduce the risk of releasing changes to the application.

Make your boss choose your priorities, that is one of the roles of a manager.

1. Add scenarios based on team knowledge and work with your boss to rank them by risk.
2. Implement top priority scenarios
3. Go back to step 1

Eventually, you will get to diminishing returns and you can move onto the next task.

Answer 2

John Ruberto wrote an article some years ago on Stickyminds entitled, "Is 100% Unit Test Coverage Enough?". The article can be found here:

https://www.stickyminds.com/article/100-percent-unit-test-coverage-not-enough

In it he presents the argument that there are different kinds of coverage. One could cover 100% of requirements, but that doesn't include edge cases a tester might look for during Exploratory Testing. It doesn't include security tests or UI tests. It doesn't cover unit tests. Now we're up to 5 passes over the application, or 500% testing. We haven't even covered the explosion of tests that configuration testing will bring.

Your boss is throwing out a number. What he actually wants is an assurance that the thing that got built is going to do what it's supposed to do. Instead of getting hung up on a number as a minimum pass requirement, show him that based on your experience, you covered as much of the application in as many ways as you can think of that matter, and found this list of things that probably shouldn't happen. Based on that information, your boss or his boss will have to make a go/no go decision.

Lee Copeland offers a different take on the subject, including answers to the question, "When do I stop testing?" in his presentation "The Banana Principle for Testers", found here:

http://www.squadco.com/Conference_Presentations/Banana_Principle.pdf

Asking for a minimum pass requirement as a percentage of anything is Management by Numbers and takes the go/no go decision out of your boss' hands. He's trying to absolve himself of the responsibility of failure by placing the requirement on you. In a shop where people make mistakes and learn from them instead of laying blame, this type of requirement doesn't come up.

Answer 3

It's always good to use numbers to make your point-

• There are N models and sub models of iPhones (for some applications you should also count the phone's network sub-type), each with M available iOS versions and sub versions (this is not entirely accurate, some models and iOS versions don't work together, but never mind that now)

• Older iOS versions needs different automation tools.

• You can communicate over WiFi, 3G or 4G and maybe Bluetooth and USB

• You will want to test P versions back on newer iOS and iPhone versions

Multiply the above and present to your boss

Answer 4

I've spent a fair bit of time working on safety-related software. People die if the software fails, kind of safety.

The first thing to note is that we had 100% test coverage of requirements. However that didn't have to be automated, sometimes because it wasn't practical, and sometimes because it wasn't physically possible. No safety-related standard requires 100% automated testing even for the most critical of systems, so the idea of doing it for an iPhone app is pretty ludicrous.

The next thing to note is that there are various sorts of coverage which you might want for unit testing. Statement coverage is a nice start, because it guarantees that at least all code is reachable, but it doesn't say anything about whether it's been tested for correctness. Branch coverage is better, but you still can't be sure about why it branched. Condition coverage is more thorough and checks that for a comparison you've given it less than, equal and greater than conditions, but now your testing is expanding. Add combinatorial logic testing to check one of each condition true, all conditions true and no conditions true. Add boundary coverage to check for maximum/minimum value inputs and outputs. And so on.

On an average project, we reckoned on around 5-10% of our time being spent on coding, 10% on requirements, and 20% on design. The rest was testing. And that was for lower safety levels. For serious safety stuff where we had to unit test like that, we reckoned on doubling that test time.

We actually analysed this and decided to abandon function-level unit tests for most things. We found that the defect-to-cost ratio was not justifiable, and many of the bugs could not in fact manifest in the system because of limits higher up. Instead, we ramped up the amount of module-level and system-level testing, because evidence showed that this was where the majority of customer-visible bugs were found.

So your boss needs to decide why he thinks this is a good idea. He needs to define what "100%" measurements you should be aiming for. And he needs to justify a 10x increase in the time to complete your project, because that's what your estimated delivery date should now say.

Answer 5

It depends what they consider 100%, but my basic answer would be 'no' especially if these are UI level tests. Tests at this level are slow and expensive to write and maintain.

It looks like you know this already, but write tests that cover the broad paths through the application/its functionality.

Answer 6

Welcome to managing your manager. I agree with the sentiment that your boss doesn't want to hear a "No". That's not quite verbose enough. But, the answer is almost assuredly a "No".

What you need is a way to communicate how and why to limit your test scope. Even if you tested all the permutations of every make-model, screen-size and OS, you will soon see that oh, there are different version of OS's and then oh, different browsers and oh those browser version. Oh, wow, now, it's next year and you have a new job because you missed the bug in your other product.

So, really you need a risk-based approach. Knowing your test variables and what they mean to your likelihood of finding a bug is crucial.

Find out what your customer-base is using. Someone should be monitoring this or collecting. And if you haven't rolled this out yet, get some consulting from a company like Perfect.

Speaking of Perfecto, you will need to vary your tests in your test automation by desired capabilities or something like that. If you have to write different automation for each of your mobile device profiles in your test bed, then forget about it. Use a visual context or make sure the site or app you are testing is testable so that you can actually hook into the DOM.

But, more than anything, decide where the risk is and have confidence that you are limiting your test scope appropriately so you can find the bugs instead of buying the magic beans of infinite test coverage.

Answer 7

I just wanted to write an update to my original question, as its been about 7 months and a lot of things have changed since I penned this question.

I decided to take a risk assessment based approach. As my manager at the time really doesn't like the word 'no' this seemed like the best option. If I could categorically prove that certain functionalities were not as suited to automation he was was more open to listening to that function not being such a priority in the automation process.

That seemed to work for me like gang busters.

What helped me influencing his decisions was using a matrix from this seminar from Angie Jones (former senior automation tester at twitter and now works for appitools).

If I had a discussion with my peers (manual tester, BA and Dev) using this matrix, it gave a holistic view and cut out the guess work - everyone was on the same page.

Then I could present to my manger the findings...resulted in him NOT stressing me out with the 100% coverage ear ache!