12

We have been developing wifi/2g boards that communicate to a server and store data.

This project has been running for several years already, and we are constantly updating both boards' firmware and server software.

Every now and then something fails, and most often detected by a client, when it's too late and that failure caused something worse.

I've conducted many tests in a controlled environment, and every time I think of a new test I do it as soon as possible, but lately I'm running out of ideas on what to test.

The boards are quite complex, and uses several chips for different purposes, so the testing involves hardware tests as well as software. I'm looking for a better approach to finding better testing methods, because obviously I'm always missing something.

I'm not suggesting that I'll be able to predict every possible case (impossible), I'm just looking for some guidelines on how to improve my tests.

7

It's difficult to answer without more details, and as you say it's a complicated system. But there are some things you should try nonetheless.

One thing you can try is doing root cause analysis to see where do bugs come from and than focus testing there., especially if more than one bug was originated from an RCA.

A good way to do that is asking the 5 whys, but be warned that there is a good chance you will end up with a procedural answer (not enough design, code wasn't reviewed or something similar) that doesn't help in locating vulnerable areas for testing.

Another thing worth trying is collecting information about the bugs in a table and trying to find correlations in things like sub module, developer, dates, programming language or environment. I am not a big believer in defect clustering as a catch all term but it worth a try.

  • 1
    I agree with most of your points, but please don't use bug tracking info to correlate to individual developers. It's too close to measuring performance and as soon as you do, the nature of your data will change as developers look to avoid blame. – c32hedge Dec 11 '17 at 17:10
  • If I read the question correctly that’s retroactive and a long time has passed since the developer has written the code. But you are right, that’s quite desperate and ineffective in more than one way – Rsf Dec 11 '17 at 17:13
  • If a specific developer has bad code management habits, and is committing sloppy, or un-reviewed code to the trunk/tag ... then why not have that discussion? – Jeremy Kowalski Dec 11 '17 at 18:12
  • 2
    Because it should have been caught beforehand, because it’s rare, because usually no one person is responsible for big chunks of code without anyone reading it and because it will usually point to bad habits in the whole team – Rsf Dec 11 '17 at 19:12
4

re:

Every now and then something fails, and most often detected by a client, when it's too late and that failure caused something worse.

I've conducted many tests in a controlled environment, and every time I think of a new test I do it as soon as possible, but lately I'm running out of ideas on what to test.

I feel that the first (real world occurances) is the better guide to better tests rather than the 2nd (your thoughts). Not that the tests should be only at the end of the process but if that is where failures show now, work your way backward from those failures to see how early in the process you could detect those sort of errors. with tests.

3

You obviously will never be able to test everything (because the number of things you could test is infinite), but you can take a risk-based approach to prioritizing what you test. For example, if you've done a risk assessment of your product, you might make sure to spend a lot of time covering cases that could lead to the most serious failures.

I also see that you've tagged this manual-testing. You could also improve your effectiveness by looking for ways to automate more of your test suite, especially since you mention constant updates. This will give you a good "safety net" to ensure changes don't break things you had previously tested. Even if you're testing systems involving hardware, there are techniques you can use to automate. Googling for "hardware in the loop testing" may yield some ideas--sorry I can't speak much from actual experience here but this is something we're hoping to do more of in our hardware/software systems.

In terms of improving your actual tests, there are quite a few books on test design and test techniques. I haven't read the first one in the list yet, but here are some helpful resources:

3

Automated regression testing. Manual testing, as you learned, just does not scale.

You should have tests for when things go as expected (happy path), and when some problems are encountered.

Also, if you can detect some common patters for problems/errors, your test can check for presence of those patterns.

Once you can trust your regression tests, humans, instead of boring repeatedly manually testing same steps, can do more exploratory testing for situation which are not covered by automated test (and automate some of them).

2

Are your tests primarily functional? Do you perform any unit level testing? If your problems are appearing in complex non-predictive business user workflows then you will end up just chasing your tail trying to script and assess every functional outcome.

Your systemic issues may be related to a few other areas:

  • Configuration management
  • Environment management
  • Poorly scoped or incomplete test data

I'd recommend reviewing any unit testing you have in place. If you don't have unit testing then I'd start by working with your development team to close the gap.

Next review your regression test bed. Do all of your tests always pass? Are you running the right tests? Have you considered taking a risk based approach to test selection? If there has been no change to an area or feature, why run the test again? Are you spending too much time running tests that are not providing you with opportunities to find issues?

How do you handle negative testing?

Finally assess the other areas that are not directly related to test or test execution. Environment management and system configuration could be part of the problem. If your testing environment is not configured exactly like production you may be getting false positives during your regression testing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.