What network-connectivity scenarios should be tested for a generic cellular IoT device?

Question

I'm working on a variety of cellular IoT products. I have developed a sense of which parts of cell stack tend to go wrong, which guides my test cases:

• SIM card may or may not be present, therefore test the device in both scenarios
• Network APN may change; therefore test how the device responds when a network connection is attempted using the wrong APN
• SIM card may not have been activated, or may have run out of data

The above scenarios are mostly focused on SIM issues, but I'm also interested in catching other issues: can the device connect to a tower in worst-case signal areas? What happens if the device is connected to one tower, but is in motion and leaves that cell tower?

So my question is: for a purely abstract, general-purpose cellular IoT device, what edge cases are worthwhile to include in a test plan for network connectivity functions?

Am I over-estimating the chance of issues due to cell-tower handoff? Are there other issues that I'm missing?

Now, I do understand that any testing in general will be dependant on product details. However, I also suspect that there's a large set of tests that should be shared across various IoT devices; this is what I'm attempting to extract.

This list of test scenarios should use the broadest set of assumptions possible: * Battery life or status in general is unknown * It's unknown where in the world the device will be operating * It's unknown what type of cellular modem is in the device (i.e. LTE vs 3G)

In my judgement, it is possible to build a mostly-objective list of cellular test cases for this scenario.

Edit: per the comment suggestion, I have cross-posted this to dsp.stackexchange.com. I may delete this question on sqa.stackexchange if the other question gets an answer. Unfortunately, the stackexchange-meta-discussion on crossposting is far too contentious to draw any real conclusions from.

Edit 2: dsp.stackexchange is threatening to close the question for not being DSP enough. Thanks, gatekeepers.

Answer 1

For a purely abstract, general-purpose cellular IoT device, what edge cases are worthwhile to include in a test plan for network connectivity functions?

It seems like you're trying for a universal test approach for similar IoT devices. Without being too general, here are some tips that might help:

One approach that's especially useful for IoT devices is mapping out their state transition diagram. This helps when defining a test strategy that covers the whole life-cycle of the device, and also helps you systematically identify different edge cases. Mind map software is great for this kind of activity.

Another way to identify edge cases is to consider Security, Performance, and Reliability concerns within each of the following categories:

(I've thrown some examples in based on common issues I've seen in IoT devices)

Hardware

• What hardware factors negatively affect battery life?
• What happens during power on/off periods, and what happens when those periods occur during important events?
• What happens during network on/off periods, and what happens when those periods occur during important events?
• What are the device characteristics when the wireless signal is significantly degraded or attenuated (e.g. does it crash? chew up the battery life?)
• Am I over-estimating the chance of issues due to cell-tower handoff? Are there other issues that I'm missing? Probably not.

Software

• Are the system logs expressive enough to help diagnose issues / crashes?
• Is it easy for you to get system logs etc.

User and/or data source

• Is hardware and software status easy to obtain?
• Is hardware and software status history easy to obtain?
• Can the device be reset easily, and does the reset state make sense? (e.g. doesn't make the thing practically useless)
• What can the primary user and/or data source do incorrectly at every defined device state?
• What can the primary user and/or data source do incorrectly at every defined device transition?