Okay, so I know we can't test every scenario possible, it's just not possible. But I want to get any ideas for making tests more comprehensive and getting the QA team to find problems before the code hits production.
We have a sizable and very configurable system and we're trying to update some major functionality with it. The problem is that every time we try to go to production something goes wrong and we hit the panic button and restore from backup.
We are now being asked why these problems didn't show up in QA or our unit tests and don't have any real good answers.
We currently have:
- a build server that runs after ever check-in,
- many unit tests (2000+ tests with over 60% coverage), but there's room for more,
- a QA environment that is similar to production (3 servers instead of 4 and older hardware, but same config and OS) however it does not have as much data as production and what it does have can get out of date,
- several in house people that we can have spend hours on QA,
- and extensive error logging.
Are we just going to have to deal with the fact that we can't find every thing that could go wrong or are there techniques to make it more likely to find the problem before end users see it?