Is it good practice to combine my Test and Prod BDD tests in one Framework?

Question

This question has been bothering me lately, more from the fact that in the past this is something I would never do, yet in the past I wasn't writing BDD tests. Since incorporating SpecFlow into a Test Framework to use for verifying functionality in my Test environment I saw some use for those test scenarios to be used in verifying Production deploys. The current Framework verifies some of the Business Use Cases and is run whenever we push a new code package to the Test Servers. Some of those Use Cases I have been running against Production whenever we do a code push to there, but in some ways this seems wrong to me. I don't want to have to rewrite tests in a different framework, but I also feel like it would be better to have confidence in our deploys and its simpler to be able to do my checking with one framework. Currently the tests are divided into two separate frameworks but for ease of use I want to combine them, but am not sure its a good idea or the proper thing to do.

There is really only me who can Test/Check code, sometimes it's hard to bounce these ideas and thoughts on improving the tests off other people so usually I muse about it for awhile and then go ahead. I built up the Framework to relieve myself of having to do some basic, repetitive Use Cases so I can concentrate on other, more detailed Use Cases. We do get value from the SpecFlow tests, and have found serious issues in both Test and Production so there is buy in from my Managers as to the value of having this framework in place.

If you are running SpecFlow how do you handle your tests for Test and or Production? Are they the same? Is there something different I could be doing to add to the value the Frameworks are providing?

Answer 1

Whenever possible I try to have the same solution for testing in my test environments as in my production environments. It isn't always possible, but when it is there are many benefits of this approach:

• You are sure the tests are the same, so all the time you have put into updating, maintaining and improving those tests for your test environment also gets put to work validating your production environment.
• You don't have to maintain two different approaches.
• Larger suite of tests run against production can give you higher confidence in the deployment.

I'm curious, what are the downsides you forsee with using a single framework?

The only potential issues I can see with running a large amount of automation against your production environment are:

• Creating load on your production servers or adding extra hits to usage data
• Potentially modifying data that should not be updated in production if you have forms that update data in a database
• Harder to single out a specific machine to test against unless you pull it out of rotation and hit it directly

Answer 2

From a pure technical design viewpoint, a single solution would most likely be better.

This is a case where the real answer depends on what the impact is if you run your "Test" tests in production.

For example, if you are testing Facebook and creating new test account from scratch and then modifying it, then you could probably run all your tests in every environment.

If you are testing a banking system then running those tests in production would most likely get you fired.

So depending on how confident you are that your mechanism for not accidentally running non production tests in production, will determine if they need to be separate or not.