In the embedded world some (unit) tests have minimal value when they run as standalone test, like unit tests should. We have many tests that run on the embedded device, as part of the embedded operating system but with most of the surrounding functionality disabled. The scope of those tests is very "white" and close to real unit tests, but the environment is very "system". How would you call such tests ? Who do you think should be responsible for them ?
2
-
Are you referring to diagnostics? I think of a diagnostic as a test that exists to help diagnose problems after the product has already been deployed. I think diagnostics should be spec'ed out by whoever is most likely to use the output.– user246Commented May 24, 2011 at 13:12
-
not diagnostics, those are real unit tests- API tests, boundary values etc.– RsfCommented May 24, 2011 at 13:29
Add a comment
|
3 Answers
You say they're close to unit tests, and running on the device, but with most of the surrounding functionality disabled. So I'd call them unit tests.
You've isolated as much as you can of the embedded device, and if you're testing each class and method in isolation with as much mocking and stubbing as possible, then they're unit tests.
The fact you're using the embedded device doesn't really matter. Compare with writing code for a PC and running unit tests on a PC. So what? They're still unit tests.
Developers are responsible for writing and maintaining unit tests. They should be runnable by anyone, ideally you should get it done as part of continuous integration if possible, but this may be a challenge as it's not always easy to hook up a device to the CI. Otherwise you should make it the responsibility of developers to run them to make sure their new/changed functionality doesn't break anything. QA/testers should be responsible for confirming the unit tests all pass before accepting the new code.
We're in a similar situation (with Symbian). We use a unit test framework that runs on the device and we mock and stub as much as possible. At first, we ran our unit tests on the actual device, running after a normal bootup. Our CI made the installation file for us.
Next, we started running them on the emulator, after a full boot up. This meant we could include the run and results as part of our CI loop for every commit (it also calculates code coverage).
Next we ran them with the emulator booting directly into textshell mode, so most of the environment isn't up and running. It's also a LOT quicker that way. In fact, when we first ran in textshell, some unit tests failed because we discovered some dependencies that hadn't been stubbed; in textshell they weren't available.
We called those the BIST : Built-In Self-Test.
The main "customer" from our viewpoint was manufacturing. They needed a suite of tests that they could run to verify that the hardware coming off the end of the line was good. The development team was responsible for implementing the tests, manufacturing was responsible for defining and running the tests.
-
I wrote BIST years ago, and they are a totally different beast... As you said BIST is meant to give PASS/FAIL results for malfunctioning hardware– RsfCommented May 24, 2011 at 13:30
-
@Rsf: Then I'm not clear on what the purpose is? Given your description, I'd still call them a BIST, even if they are not explicitly for manufacturing.– Peter K.Commented May 24, 2011 at 13:43
Depending on the project I've worked on, these tests are called Unit Tests on Target or HW/SW Integration Tests. Their purpose is to test hardware features such as watchdog or ISR. Some tests deserve their own category, such as Boot Tests.
