8

The QA manager in our organization is demanding that our dev team, which I am the manager of, do the smoke testing after we set up the testing environment and before the QA team begins testing, i.e. he wants us to sign off on the environment and to say that it is stable enough for testing.

I was wondering if this is a common practice in many organizations. If not, what are the more common practices.

  • Is the manager want it automated or manually? If automated, then CI with automated deployment containing hooked up tests helps a lot or solves this fully. If he/she want it manually, then it is another question. But all in all, which tests are considered part of this particular smoke test defined by QA and Dev and depends on the previous experience, meaning that what type of defects blocked the testing on the environment. – SayusiAndo Apr 14 '15 at 12:26
  • Just curious. Do you guys write unit tests? – saifur Apr 14 '15 at 13:50
  • @saifur - Yes, we do, but unit tests are only half the story. Our application has many dependancies (WS, DB, etc.) that are out of our control, and most of the time, the smoke tests fail due to these dependancies. – Elad Lachmi Apr 14 '15 at 13:59
  • A small number of deployment-validation tests to exercise the dependencies sounds like a good idea. However, I agree that needing the developers to do functional testing of each module sounds like overkill. – Vince Bowdren Apr 14 '15 at 15:12
  • 2
    We had a similar policy in the past, where after a release not only would Dev do the deploy to make sure the build worked but they would smoke test by checking areas they did not work on. That provided them valuable experience in other areas and also made sure no one had their blinders on for possible issues. – MichaelF Apr 14 '15 at 15:29
12

I cannot speak for the industry in general, but at places I have worked, developer-initiated smoke-testing is a common practice if builds delivered to QA tend to be unreliable.

I had a job as a test lead on a team that required developers to smoke-test the builds. At first it was a painful process, taking up most of a developer's time for an entire morning. Eventually we automated the deployment and wrote some automated smoke tests. That freed up some developer time while giving the QA team an indication of whether testing the latest build was a good use of their time.

The "developers do the smoke tests" policy forced developers to pay more attention to the quality of the code they were checking in, which of course was the whole point. Over time, the build quality improved and the smoke test became less necessary.

  • Ok, I can see the logic. Im not happy with my team doing what they (and I) consider QAs work, but I now understand the benefit. – Elad Lachmi Apr 14 '15 at 17:29
  • 1
    The work of everyone is to assist in the production of high quality software. The sooner errors get into the hands of developers, the better it is for everyone. Developer initiated smoke tests help prevent "Infinite Defect Methodology" The schedule was merely a checklist of features waiting to be turned into bugs. In the post-mortem, this was referred to as "infinite defects methodology" Source, scroll to #5 – corsiKa Apr 14 '15 at 18:21
2

I agree with @user246 and @Niels van Reijmersdal, but it grew longer than comment. :-)

If developers are allowed just "throw code over the wall" to QA, even if basic smoke test will fail in QA environment, they will have no incentive to fix the smoke testing process - and eventually automate most of it.

Goal is not kick the can down the road and make the failure someone else's problem - goal is to make most productive use of the time of every member of the team. If smoke tests fail, developers should be able to find cause faster - so it will be better use of team's time that QA manual testers trying to figure it out.

  • 1
    I really like this answer. It seems to imply that if the smoke test the developers run is big enough that you think QA should be doing it, you're not really doing a smoke test anymore. – corsiKa Apr 14 '15 at 22:51
1

Sounds reasonable that you deliver an environment which at-least starts. Smoke tests often just consist of a check that the application starts and some basic functions work.

I think its a best practise is to setup a continuous integration environment which:

  1. Builds the application
  2. Runs the unit-tests
  3. Runs some integration-tests
  4. Runs some GUI tests
  5. Deploys to testing environment

After step five the QA-team can start working on it.

Get their help to setup the integration and GUI tests that they need for it to be able to start testing on it. I would let the QA-team define the Smoke test and let the Dev team automate it.

  • Thank you for the quick reply. Of course we set up a functional environment. The problem is that we have 35 (and growing) different modules and he wants us to go in to each one and check that the basic functionality is there. Even for modules, which we have not touched. This seems to me like something a QA engineer should do, but I`m biased :) – Elad Lachmi Apr 14 '15 at 12:03
  • 3
    Sounds like something an automated test should do, don't waste man hours on repetitive tasks. Test automation is the future :) Do wonder why QA is asking you this, probably they had situations where modules where not "touched", but they didn't start. Try to solve that problem instead. – Niels van Reijmersdal Apr 14 '15 at 12:08
1

Continuous integration server (Jenkins, Teamcity etc.) with unit-tests and automated functional smoke test test run before build is our choice in perfect work flow. And we have some checklist for developers that they should execute before their commit to repository at initial state before we do not have Continuous Integration server.

0

Where I work, a developer is expected to get their new feature deployed to the Dev environment and make sure that it and the platform are both still functioning at a very basic level. We had a period where quite a lot of new work checked in and deployed to Dev did not work at all, and got pushed straight back pretty much right away. I think that level of smoke testing from devs should be expected, as a handover period of the feature to QA.

The rest should ideally be covered by CI and any new, non covered functionality be picked up by the QA team, once the feature is in a testable state.

This means the Dev on the feature tests a few of the most basic functions of that feature once deployed to Dev to ensure its ready for the QA team to pick up, the Dev should absolutely not be spending much more than about twenty minutes doing this sanity check for most work: once it's deemed the feature is running then QA can pick up from there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.