How to choose hardware for testing computationally intensive Android software?

Question

I work on mathematical modelling software, which runs on a variety of platforms. The product is a library, not an app.

For Android, I've found it easiest to re-use the command-line test harness that's used on UNIX-family platforms, and run that via the Android Debug Bridge. That's an ingenious piece of software that gives you access to a UNIX-style shell running on an Android device. I put my libraries, executables and test data on the device, under /data/local/tmp (an area where the shell has full control of the file system) putting them there with "adb push". I automate testing with shell scripts, some running in the adb shell, and some running on the Linux machine I use as a development host.

There is a lot of testing, and it takes a long time to run, even on reasonably powerful devices. It also needs a lot of RAM: as of early 2023 at least 8GB available after operating system overheads, and at least 128GB of storage for test data. The testing runs overnight, so methods involving unplugging and re-plugging devices are off the table.

In my case, it is sometimes necessary to run testing continuously for more than a week, so battery capacity is not really relevant: the hardware has to sustain testing on its power feed, rather than draining the battery.

In my case, wireless communication is not used, because the test data is quite confidential, so we prefer to have the devices in airplane mode and do everything over USB. However, different strategies may be appropriate for apps, rather than libraries.

What are the principles for selecting hardware to get this job done efficiently?

Answer 1

As of April 2023 I've found a better solution in Qualcomm's line of Mobile Hardware Development Kits. These are single-board computers, containing high-end Qualcomm SoCs. They're really intended for hardware development, but they work fine for software testing.

The latest device has the Snapdragon 8 Gen 2 SoC. I have the previous generation, with the Snapdragon 8 Gen 1, 12GB RAM and 256GB storage; the oldest model available has the snapdragon 888.

The key feature of these devices for me is that batteries are optional. They come with a mains adapter (100-240v, 50-60Hz) and run happily from that without a battery. I therefore can't run out of battery, or damage the battery, and can make full use of the Cortex-X series performance cores in these SoCs.

They have a few drawbacks, but those are less important for my purposes than their performance:

• I had to update their firmware to get access to all of their storage. The instructions for this are vague, but the correct process is easy to summarise: do it from Linux, and run the update script with sudo. The developers of the script are hardware engineers at Lantronix, the distributor. They do all their work on Linux, and appear to assume that anyone working with these devices will use root powers all the time.

• As the devices have no battery, their real-time clock is not battery-backed. If they aren't connected to a wireless network of some kind, they lose track of the date and time when powered off. Since I run them in airplane mode, I have to reset the date+time occasionally. As they come with a working su the easy way to do this is from the adb shell with the date command.

• Buying the add-on touch-screen makes setting up Android far easier.

• You need to provide a case yourself: neither Qualcomm nor Lantronix provides them. You'll need stand-offs to mount the device in the case, and for no plausible reason, the mounting screws needed for the main board and the touch-screen are different sizes, at least on the model I have. You'll need M2.5 and M3 parts.

• They are not cheap, at about US$1,000 for a board and touch-screen, but to me, they're worth the money.

Answer 2

Basics

The methods I've explored for this have been:

• Canonical Anbox Cloud lets you run Android apps in containers on Aarch64 Linux, either on-premises, or on cloud services that offer Aarch64 instances. I didn't go for this, because you're running on a Linux kernel, rather than an Android one, the CPUs you can get on cloud services aren't as fast as leading-edge 'phones, and the support is pricey.
• Actual Android, on Android devices. I did this with devices based on Qualcomm Snapdragon 835, 845 and 855 SoCs. The problems were that they were slow - the amount of testing is constantly growing - and that the amount of RAM available was marginal - the record for "biggest test case" is constantly being broken, for good reasons.

Power problems with ARM Cortex-X series

When the Snapdragon 888 appeared, it looked like a solution to my problems. The Cortex-X1 core is much faster than the Cortex-A7x series, and the SOC supports up to 16GB RAM.

The difficulty, as I learned from costly experience, was that none of the Android 'phones with a Snapdragon 888 could accept power from a host computer fast enough to keep up with the consumption of a Cortex-X1 running flat out. Or rather, they might be able to do so, but they could not talk to a host computer over USB at the same time.

• A device with two USB-C sockets and its own 65W charger could only accept the full charge rate on its "primary" port, and could not communicate via its secondary port at the same time with a host computer. If I plugged the charger into the secondary port, the primary worked fine for communicating with the host, but the secondary could not accept enough power to keep the battery from draining and the device running out of power.
• I tried a device with one USB port and wireless charging. The manufacturer's technical support were sure it could communicate over USB and take charge wirelessly at the same time. They were wrong: communication went down when I put the device onto the cradle, and came back when I took it off. I did at least get a refund on that device.

Best option with consumer devices

As of March 2023 the best bet seems to be a device with a Snapdragon 870 5G. That's the most modern Qualcomm SoC without a Cortex-X-series core. The one I've tried is just about fast enough, it has just about enough RAM at 12GB, and storage is no problem now that 256GB is standard for high-end 'phones.

As of April 2024, the fastest Qualcomm SoC without a Cortex-X series core is less clear. The Snapdragon 870 5G is still the latest in the 8-series, but it has Cortex-A77 cores, which have been superseded several times. The Snapdragon 7 Gen 3 has Cortex-A715 cores, but less memory bandwidth; benchmarking would be necessary to tell which is better for testing a given application.