Test for unknown issues

Question

We have been developing wifi/2g boards that communicate to a server and store data.

This project has been running for several years already, and we are constantly updating both boards' firmware and server software.

Every now and then something fails, and most often detected by a client, when it's too late and that failure caused something worse.

I've conducted many tests in a controlled environment, and every time I think of a new test I do it as soon as possible, but lately I'm running out of ideas on what to test.

The boards are quite complex, and uses several chips for different purposes, so the testing involves hardware tests as well as software. I'm looking for a better approach to finding better testing methods, because obviously I'm always missing something.

I'm not suggesting that I'll be able to predict every possible case (impossible), I'm just looking for some guidelines on how to improve my tests.

Answer 1

It's difficult to answer without more details, and as you say it's a complicated system. But there are some things you should try nonetheless.

One thing you can try is doing root cause analysis to see where do bugs come from and than focus testing there., especially if more than one bug was originated from an RCA.

A good way to do that is asking the 5 whys, but be warned that there is a good chance you will end up with a procedural answer (not enough design, code wasn't reviewed or something similar) that doesn't help in locating vulnerable areas for testing.

Another thing worth trying is collecting information about the bugs in a table and trying to find correlations in things like sub module, developer, dates, programming language or environment. I am not a big believer in defect clustering as a catch all term but it worth a try.

Answer 2

re:

Every now and then something fails, and most often detected by a client, when it's too late and that failure caused something worse.

I've conducted many tests in a controlled environment, and every time I think of a new test I do it as soon as possible, but lately I'm running out of ideas on what to test.

I feel that the first (real world occurances) is the better guide to better tests rather than the 2nd (your thoughts). Not that the tests should be only at the end of the process but if that is where failures show now, work your way backward from those failures to see how early in the process you could detect those sort of errors. with tests.

Answer 3

You obviously will never be able to test everything (because the number of things you could test is infinite), but you can take a risk-based approach to prioritizing what you test. For example, if you've done a risk assessment of your product, you might make sure to spend a lot of time covering cases that could lead to the most serious failures.

I also see that you've tagged this manual-testing. You could also improve your effectiveness by looking for ways to automate more of your test suite, especially since you mention constant updates. This will give you a good "safety net" to ensure changes don't break things you had previously tested. Even if you're testing systems involving hardware, there are techniques you can use to automate. Googling for "hardware in the loop testing" may yield some ideas--sorry I can't speak much from actual experience here but this is something we're hoping to do more of in our hardware/software systems.

In terms of improving your actual tests, there are quite a few books on test design and test techniques. I haven't read the first one in the list yet, but here are some helpful resources:

• How to Break Software, by James Whittaker. There are two other books in this series focused on web software and security, respectively.
• The Art of Software Testing, by Myers, Badgett, and Sandler
• Exploratory Software Testing, by James Whittaker
• Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing, by Elisabeth Hendrickson

Answer 4

Automated regression testing. Manual testing, as you learned, just does not scale.

You should have tests for when things go as expected (happy path), and when some problems are encountered.

Also, if you can detect some common patters for problems/errors, your test can check for presence of those patterns.

Once you can trust your regression tests, humans, instead of boring repeatedly manually testing same steps, can do more exploratory testing for situation which are not covered by automated test (and automate some of them).

Answer 5

Are your tests primarily functional? Do you perform any unit level testing? If your problems are appearing in complex non-predictive business user workflows then you will end up just chasing your tail trying to script and assess every functional outcome.

Your systemic issues may be related to a few other areas:

• Configuration management
• Environment management
• Poorly scoped or incomplete test data

I'd recommend reviewing any unit testing you have in place. If you don't have unit testing then I'd start by working with your development team to close the gap.

Next review your regression test bed. Do all of your tests always pass? Are you running the right tests? Have you considered taking a risk based approach to test selection? If there has been no change to an area or feature, why run the test again? Are you spending too much time running tests that are not providing you with opportunities to find issues?

How do you handle negative testing?

Finally assess the other areas that are not directly related to test or test execution. Environment management and system configuration could be part of the problem. If your testing environment is not configured exactly like production you may be getting false positives during your regression testing.