When there is a new build we perform smoke testing,in few projects they call it as sanity testing. Can someone please elaborate on the difference between the both?
Add a comment
|
8 Answers
Just to muddy the waters even more I'm going to throw in "it depends". If you're looking for a practical answer within your own company then I'd ask them, as they are terms that change for different domains. Sometimes they are the same, sometimes they differ, and often they are defined at the whim of whomever writes the tests.
Stuartf's answer is as close to an "offical" definition as you will get. A little history and context will help display the difference:
Smoke tests refer orignally to hardware testing, wherein a device would be plugged in and if it started to smoke then it was unplugged as no further testing was required. It doesn't work, there is no point in testing it.
Sanity tests are, traditionally (in other realms of testing) tests that establish whether or not a result could be true, rather than if it is true, that is to say if the output is "sane". If you put "9" into the function "squareroot()" and get "badger" you have failed a sanity test. If you get, say, a positive number less than 9 then you have passed it, even if the number is not "3". This is why they refer to tests that ensure that the system is working enough to be tested - it is giving a sane response to basic tests, therefore it is not a waste of time to run full functional tests.
The confusion comes when lost in the complexity of software - at what point is the system testable? How can we see if smoke is coming out of it when it only breaks when a certian section of the code is run? It differs from project to project, evidently. Also in order to do a smoke test you need to use your testing framework (manual or otherwise) to actually check if the system is running and responsive; so in many cases they ARE a subset of functional tests. So the definitions are different, but are (often erroneously) used interchangably on a practical level. Hopefully knowing the context will give some meaning to the words, even if they both reduce (practically and contextually) to "give it a kick and see what happens".
Their purpose is similiar in software testing, to save time on running a full regression suite and finding out that the system falls over because of a base error. The reason being that a basic error in the system has to be fixed, and that fix affects the system, and the testing should then be restarted, so the quicker that larger, high priority errors can be found the better. For example a web application may be spidered for 404 errors and broken links before long, slow, functional tests are run.
Hope that's helpful!
-
+1 for an EXCELLENT analysis of the "It depends" aspect of things. On a side note, for the hardware smoke test, I've experienced this although, in our case, it was a flame test... old Leading Edge D2 PC, plugged it in, capacitor in the RAM circuits exploded (literally) and caught the motherboard on fire... Yup, that one doesn't work... Commented Jun 21, 2011 at 14:45
-
2I envisiage a smoking wreck with a "no further testing required" sticker on the side :). I've suffered a capacitor explosion myself in a speaker system, it caked the inside of the casing with smoking dialetric, so I know from whence you come! Many thanks for your kind words Commented Jun 21, 2011 at 16:14
-
Your mental image is accurate... mix in the smell of chemical fire-extinguisher and scorched silicon and you've pretty much described the situation... Commented Jun 21, 2011 at 17:27
-
1
-
1Heh you guys are making me remember an old article on new solid state devices that appeared in an april issue of popular electronics long ago (I'm ancient). Three were applicable to this discussion.. SSCG Solid State Confetti Generator (25uF electrolytic 5v cap, placed across 120vac source.. good for one use only. NED the Noise Emitting Diode (Place a normal LED across 120vac source, makes a loud noise, once.) and DED -Dark Emitting Diode, what you have after a NED has been used once. Commented Jun 23, 2011 at 4:45
Sanity and Smoke testing are conflated. They are not the same and they have a different purpose!
Smoke tests determines whether it is possible to "continue" testing and sanity tests determines whether it is "reasonable" to test.
A smoke test should only ensure that application launches and that all the interfaces are responsive. You do not test a subset of functionality in a smoke test.
where as
A sanity test should ensure that the application is in a state that it is ready for more rigorous testing. This is done by executing a small subset of you functional tests.
As Joe suggests, they are often used as a gate to make sure that time isn't wasted on testing a build that is fundamentally flawed.
-
I have also seen the 'sanity' test done as a quick test (in environments without good automated regression suites) ala your description, a random sampling of functional tests across the whole product, to ensure that a recent patch or 'hot fix' didn't break anything. That's a very different thing than 'is it good enough to test' it's closer to 'we don't have time to retest it all, does it look like nothing suddenly started behaving in a crazy way.' Commented Jun 23, 2011 at 4:50
In most shops, there is no difference. Often, they are two names for the same process.
In both cases, they are a subset of all your test cases, covering the main functionality of your system, to ascertain that the most crucial features are working, but not bothering with finer details.
Many times the smoke test/sanity test is used as a gateway, through which the build must pass before it's deemed worthy of more in-depth testing. If the build fails this test, it goes back to the dev team for fixing. If it passes, the test team then digs in.
answered Jun 21, 2011 at 11:45
-
I concur with Joe, I've heard both used and when they are it's usually to refer to the same thing.– MichaelFCommented Jun 21, 2011 at 12:18
-
1They are not the same thing, unfortunately this is common mistake in software quality assurance. They do share similar characteristics, however, they serve a different purpose.– stuartfCommented Jun 21, 2011 at 12:52
-
We need a wiki page to cover all these terms and their most common definitions, because this question will come up again.– user246Commented Jun 21, 2011 at 13:05
-
Looking at the glossary in a textbook (The Testing Practitioner by Erik van Veenendaal), for Sanity Test it says "see smoke test".– Tom77Commented Jun 21, 2011 at 15:27
-
1In most shops, the terms are used interchangeably. In some shops they are distinct terms, as in the case of stuartf. Our profession has no real rules for terminology, and even the common usages are violated by some shops every day. I've encountered almost as many different phrases that mean something like "a gate to make sure that time isn't wasted on testing a build that is fundamentally flawed" as I've encountered companies. (Not to mention: strazzere.blogspot.com/2010/04/bug-by-any-other-name.html ) :) Commented Jun 21, 2011 at 16:39
To remember easily,
Smoke test - you just test all pages are accessed without errors and whether input is submitted and you get results. You do this across the application.
Sanity test - You take 1 ore 2 modules. probably the more no. of fixes it has or the important one and test it deep by going through all functions.
You understand that you do this only to accept the build for further testing, ie. regression, system, etc.
answered Aug 11, 2011 at 12:24
Smoke testing is normal health check up of a build before starting the in-depth testing.
Sanity testing is to verify whether requirements are met or not, checking all features breadth-first. Depth of sanity testing is more than smoke testing. Sanity testing helps in identifying whether the test case results are consistent or not.
Some people consider both smoke and sanity testing as same.
Other books which discuss the difference:
- Software Testing by Limaye
- STRUCTURED SOFTWARE TESTING by Arunkumar Khannur
- Software Testing: Principles and Practice by Srinivasan Desikan, Gopalaswamy Ramesh
You can also have a look at Regression / Smoke / Sanity
As the name suggests->
Smoke testing-> Here you check your entire system and check if it works fine or has any issues (smoke). This is normally done on stage before deploying the code to prod.
Sanity Testing-> Here you do a last minute check. We only test the major functionality and see if the sanity is maintained.
Hope this helps !
Basically smoke testing is carried out once the product have been developed to ensure all the functionalities are working fine, whereas sanity testing somewhat for we ensure that the bugs/errors have been removed after the recent changes made in the code. I've been working on this topic and shared a guest post on it. You can get complete difference between them in this article. Hope this question might be useful to your question.
sanity testing means we have to go through every main part in detail we can avoid parts who have low severity and in smoke testing we have to check each and every part in cursory way...each and every parts should be tested
