Smoke test vs Sanity test vs Regression test vs Re-Test

Question

After a few years in software testing journey I have faced several testing types but when deal with manager's question like "Do we have to do Smoke/Sanity/Regression/Re-Test to ensure quality of application?" I still have some confusion.

Two type that I very clear is Re-test: mean that when bugs are fixed we do this type to ensure that bugs do not happened any more and Regression test: I test (if can) all previous features that have worked very well before to ensure that bugs fixed/new deploy feature do not affect those features.

For Smoke test this type need to run before we start any testing with build deployment and this test is just a "surface" level.

But how about Sanity test? I know that this type is for "check the new functionality / bugs have been fixed" or "to verify the "rationality" of the system in order to proceed with more rigorous testing" so what is its difference with Re-test?

And as my knowledge we should execute those tests in order like: Smoke -> Sanity -> Re-test -> Regression, is it right? Because I see in this article it says- "Re-testing is done before we start sanity testing".

Answer 1

As per my understanding article is right. Let's understand the process from the beginning with respect to the timeline.

Starting Point: Once the software is ready (hopefully unit & integration testing is finished in the sprint, QA will have already been involved and dev/QA are in sync)

Smoke Testing: Build is ready & Smoke testing starts by installing an application which will determine if the install was successful and the application is functioning. It will take at least 30 minutes unless the app is humongous and Test team will be doing smoke testing or parallel automation to verify and get the verification to 30 min or less.

Sanity testing: it is the areas to perform exploratory testing(high level) to verify that the rest of the application which should have been untouched is indeed "untouched" and functions the same this is the best way to find the bug in the untouched area & report the bugs if any otherwise move to next stage.

Re-Testing: As you said Re-test mean that when bugs are fixed we do this type to ensure that bugs do not happen anymore

Regression Testing: Regression testing is next. Regression is detailed testing targeting affected areas that would likely have been modified in some way which could lead to bugs. mostly run automation regression suite.

Functional testing: Functional testing of the new expected new functionality happens. This is to ensure that the build is actually doing the improvements that were expected. No point in doing a whole bunch of another testing if the build doesn't even do what it's supposed to be improved in doing.

Note:- Functional and Regression may be interchanged depends on the time of the release.

Answer 2

Confusion is caused by minute differences in terminology which can be mostly ignored in real life.

Smoke testing and sanity testing are first manual, quick-and-easy tests you perform, to see if something obvious is horribly broken (and more testing would be waste of time), so you can fail the build quickly and save time.

Regression testing - should be automated, and might include various levels of tests (like smoke and sanity). It starts automatically so you (tester) are not involved in timing it.

Re-testing - is manual testing of the cause of failure of the recent build, if you do not have automated tests for it. Again, to quickly fail the build if fix did not worked as hoped for. If simple, do it before smoke tests. If more involved - after. Most recent code changes are most suspect, so test those early, to detect breakege early.

It is just common sense: walk before run. No rocket science.

Answer 3

There isn't really a hard and fast rule about test types. Each system or application has its own risks and requirements that will define how much testing at different levels is needed.

All to often agile testing approaches skip having a test plan and so creates this confusion because there doesn't seem to be a place for top down view.

It's actually a good idea to have a basic test plan where you identify the specific risks of your project and in retrospectives evaluate the need to alter it. Different parts of your system will have variable risk at different times.

You can run the same test in many of the different test levels but it's how you interpret and tirage the results that makes the difference.

As you go further toward the far end of the test pyramid the costs of testing increase and so should your certainly that the test will pass, that is really what the levels are about , choosing the most appropriate tests and approaches to match the risks you are trying to mitigate.

The names are just markers and as a tester it's up to you to define them for your team and maintain that definition.

Answer 4

Firstly we need to be clear about Software Build: Software Project consists of numerous source code files. Creating an executable program from these source files is a tedious task. We need to use "build" software to create an executable program, so this process is called "Software Build".

Below mentioned terminologies are commonly used in software qa company. So lets understand the same.

Smoke Testing: Smoke Testing is a kind of Software Testing performed after software build to ascertain that the critical functionalities of the program is working fine. The purpose is to reject a broken application, so that the QA team does not waste their precious time.

Sanity Testing: Sanity testing is a kind of Software Testing performed after receiving a software build, with little changes in code, or functionality, to identify that the bugs have been fixed and no further issues are introduced due to these changes. The aim is to determine that the proposed functionality works somewhat as expected.

Regression Testing: Regression testing is a type of software testing which is carried out to make sure that the fixes or enhancements to the application have not affected the other parts of the application.

Re-testing: Retesting is a type of software testing which is done to ensure that the test cases which failed in the previous execution cycle pass after the defects against those failures are fixed.