When to switch from "Debug" build testing to "Release" build testing?

Question

At my company I sometimes have trouble with getting across to developers as to why we testers shouldn't be testing in "debug" towards the end of the project. That said I'm not quite sure WHEN we should move from an e.g. MS Visual Studio "debug" generated build to a "release" generated build.

Does anyone have any ideas on this?

From past experience for example, we would have "release only" issues where the devs would say "release doesn't initialize pointer variables while debug does this by default" which would be the cause of some release build issues. They would then go back and initialize these pointers to NULL.

This article does help but doesn't have quite the answers I'm looking for:

https://stackoverflow.com/questions/420343/separate-debug-and-release-builds

Answer 1

Testing in debug mode will only let you reliably test truly surface stuff, like UIs. There's so much extra stuff loaded "under the hood" - like stack frame protection - when running in debug, that you're end-stage-loading the testers finding nasty stuff like memory allocation and stack problems (depending on what language you're using).

Any formal testing should really be done in release - the good news is that it's possible to "debug" without "full" debugging - Visual Studio can generate symbols, etc without actually enabling full debug, allowing a sort of "best of both worlds", with code which is build without the _DEBUG define.

Answer 2

Testern should not test in debug mode, but only devs should do this in their development tests.

A good testing strategrie provides the following enviroments:

1. Developement enviroment for development tests (with debugmode switchable by the devs for on/off
2. Integration eviroment for the QA to test the software under real conditions - this especially means debug mode off.
3. Live enviroment

So the devs can make theier development tests with debug mode or without it while the QA tests under real conditions. Maybe before the software goes live you want a forth enviroment for the customers acceptance test. But this depends on your company. Some can and want to afford this, some use the integration enviroment for this.

Answer 3

This will depend on what level you are testing. As you have tagged the question as manual testing I will assume you are testing functionality that will be available to a client/user. In that case you need to be testing on the release build because this is exactly what the client/user will be running.

The debug build is a system in transition. It serves use to the developer as they can step through code line by line. The debug can do things differently (as you have said in the OP). The developers should never be handing over a debug build expecting a tester to sign off as "ready for release".

Possibly early in a cycle a developer can hand over a debug build for a sanity check or precursor to a bigger release. This can help testers flag issues earlier than waiting for a release-ready build. Like I said above, I would never recommend to sign off on a debug build.