3

This question has been bothering me lately, more from the fact that in the past this is something I would never do, yet in the past I wasn't writing BDD tests. Since incorporating SpecFlow into a Test Framework to use for verifying functionality in my Test environment I saw some use for those test scenarios to be used in verifying Production deploys. The current Framework verifies some of the Business Use Cases and is run whenever we push a new code package to the Test Servers. Some of those Use Cases I have been running against Production whenever we do a code push to there, but in some ways this seems wrong to me. I don't want to have to rewrite tests in a different framework, but I also feel like it would be better to have confidence in our deploys and its simpler to be able to do my checking with one framework. Currently the tests are divided into two separate frameworks but for ease of use I want to combine them, but am not sure its a good idea or the proper thing to do.

There is really only me who can Test/Check code, sometimes it's hard to bounce these ideas and thoughts on improving the tests off other people so usually I muse about it for awhile and then go ahead. I built up the Framework to relieve myself of having to do some basic, repetitive Use Cases so I can concentrate on other, more detailed Use Cases. We do get value from the SpecFlow tests, and have found serious issues in both Test and Production so there is buy in from my Managers as to the value of having this framework in place.

If you are running SpecFlow how do you handle your tests for Test and or Production? Are they the same? Is there something different I could be doing to add to the value the Frameworks are providing?

2 Answers 2

4

Whenever possible I try to have the same solution for testing in my test environments as in my production environments. It isn't always possible, but when it is there are many benefits of this approach:

  • You are sure the tests are the same, so all the time you have put into updating, maintaining and improving those tests for your test environment also gets put to work validating your production environment.
  • You don't have to maintain two different approaches.
  • Larger suite of tests run against production can give you higher confidence in the deployment.

I'm curious, what are the downsides you forsee with using a single framework?

The only potential issues I can see with running a large amount of automation against your production environment are:

  • Creating load on your production servers or adding extra hits to usage data
  • Potentially modifying data that should not be updated in production if you have forms that update data in a database
  • Harder to single out a specific machine to test against unless you pull it out of rotation and hit it directly
3
  • I've always tended to run a Test Suite in test and that has been the most automation I had, running automated acceptance suites in Production is slightly new. Before I just used a checklist, and some Use Cases from the code package. I'm not sure about downsides, but in some ways I feel like I shouldn't be using the same tests in both environments even though I know its simpler for me. I don't worry about load, since when this is run it's late at night when usage is light or we are in a maintenance window.
    – MichaelF
    Feb 2, 2012 at 21:18
  • I have always wished I could run the same automation in production, but always had reasons I could not - most of the products I have worked on I could not modify anything in production and most of my tests ended up modifying something. In those cases, I have still had a subset of my automated tests that I could run in production, and tagged them in a way to where it is easy to run only the production tests.
    – Sam Woods
    Feb 3, 2012 at 1:33
  • This has been my experience as well, but I built up my tests this time to cover more functional areas; plus what I am testing doesn't really allow itself to manipulation or a framework.
    – MichaelF
    Feb 3, 2012 at 12:33
4

From a pure technical design viewpoint, a single solution would most likely be better.

This is a case where the real answer depends on what the impact is if you run your "Test" tests in production.

For example, if you are testing Facebook and creating new test account from scratch and then modifying it, then you could probably run all your tests in every environment.

If you are testing a banking system then running those tests in production would most likely get you fired.

So depending on how confident you are that your mechanism for not accidentally running non production tests in production, will determine if they need to be separate or not.

2
  • As someone who has worked on banking systems, I agree with you.
    – user246
    Feb 4, 2012 at 21:28
  • If the Framework is built to run only specific tests in each environment the impact can be mitigated, this is something I tried with Tags and SpecFlow unsuccessfully. Some tests are ideally suited to run in both environments and provide some Acceptance Test capability in Prod. I think part of this stems from my comfort level and these answers are making me think that I should be more than able to run them in both if I deem them capable.
    – MichaelF
    Feb 6, 2012 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.