Cloud device lab vs. in-house (physical) device lab

Question

We are considering two options for our native mobile apps (Android and iOS) testing environments: Either rely on a cloud device lab (AWS device farm) or build our own in-house physical devices lab (and possibly use with it mobile testing emulators as well). Our testing will include both automated tests plus manual tests.

While each has its own pros and cons (device cloud might give us access to more devices in smaller costs, but inhouse device lab might give us more control over our devices), we need to make up our mind on the direction to take.

Are there any things to consider when taking such a decision? Is there an always-correct answer, or it depends on different factors?

Answer 1

I've done two conference talks about building a mobile device lab for my company, part1 is on YouTube, but sadly it's in Polish only: https://youtu.be/jz-pdj2XIno

(If you would need more help I can make a CC for this video, or write a blog post about it or smth, DM me if you need)

A quick summary with a translation: There's no "best" answer or solution here.

Both in-house and cloud solutions are good or bad depending on your needs.

Cloud solutions can be very expensive. Based on our need we would need to pay around $1,000 per device per month. So for 20 devices, it would be around $240,000?

But that's just a simple example of course.

And that's because we would need those devices for our use only, because of the VPN certificates and a few other problems.

So I've created a nice report trying to figure out the solution in-house.

I've come with 3 ideas on how to approach creating an in-house mobile device lab, maybe they will help you.

In the "open one" you just simply put your all devices on the IKEA-style shelf, plug them into a power source and maybe manipulate remotely.

Anyone can access them and borrow them.

Closed device lab approach, like one we can see at Facebook. Close those devices in the dedicated rack shelf and you can automate 24/7. Remote manual access for Android devices can be done for example using OpenSTF.

Merged.

You can use your already own devices.

Manual testing at a working time (Mon-Fri) 8-16 or so. And after 16-8 and on a weekend you can run automated tests on those devices.

There's a ton of risk going with an in-house device lab.

Starting with batteries blowing up and catching fire. It's unlikely, but are you prepare for taking this risk? Do you have a dedicated room, fireproof rack-shelf, temperature measurement devices, scripts that will check your devices constantly?

And that's just the beginning.

Those mobile devices are not designed to run on a power source 24/7 for months, you can check that with devices manufacturers. That's not covered by the devices guarantee.