6

I'm looking into running tests in production that sign-up a paid user. From the sources I've seen (Google test automation conference, Microsoft Blog) it looks like testing in production (after local, and staging testing of course) is a good practice.

The reason we would want to test paid signup, is to ensure the system is working end-to-end on production servers, and immediately detect any breakage.

I don't want to boost our revenue numbers with these fake signups though. I can keep a database table of the test users, but I was wondering what is usually done to prevent this?

Or do most people rely on server health checks and log monitoring, rather than confirming the end-to-end process is working?

Improve this question
2
  • What sorts of things are required for a paid user? Do you require credit cards, valid billing addresses, etc? This should dictate the restraint or stubs you use when testing. How would creating fake signups boost revenue numbers? What's the risk?
    – Chris Kenst
    Commented Jun 1, 2015 at 23:39
  • Yes, a valid credit card and billing address is required. We use a third party service that we post the credit card info directly to, per PCI compliance. So if we create test paid signups, they would end up in this third party, which we use in calculating revenue. Fraud is a risk if we don't remove the fake users from these calculations.
    – trotha01
    Commented Jun 3, 2015 at 2:18

3 Answers 3

Reset to default
3

You definitely want to smoke test on production. If there is an issue you want to know first. Automated tests should help there. Your payment system will probablly have a different account than your staging environment, so best to double check the whole process so you can sleep at night.

As for the metrics, you need to have some mechanism to label the account as a test account so that it is not considered. For example, if the test account's email address matches your domain or some fake test domain then your report would exclude it.

Improve this answer
1
  • Thank you for the confirmation on testing on production servers. I'll have to collaborate with the revenue team to see how they want to manage these test accounts.
    – trotha01
    Commented Jun 3, 2015 at 2:29
2

Depending on where you live and whom you work for, fake money transfers on a production system could have serious legal consequences. Same for undocumented money transfers between departments within the business. Consult the legal department in writing if you have one, otherwise your boss, regarding refunds/reimbursements and the necessary documentation.

Kirby's suggestion to install a "back door" on the production system has some merit -- a transaction that didn't happen doesn't have to be canceled. But if you do that, document it and plan on some audit procedures -- who did use it when, and do you know why? Instead of string-matching of domain names, consider a flag in the database that can be set at need. Other departments like marketing are likely to jump on the bandwagon if they learn about these "freebie accounts" ...

You should still smoke-test the live payment process, but that should be one carefully planned and documented test to get maximum information out of your business transaction. Then undo it in accordance with your legal requirements.

I would never make that part of an automated test suite if money changes hands for real, even in tiny amounts. One concern among many, the automation would need to enter the account credentials.

Improve this answer
1

We have same situation in my project. As a tester we just navigates to payment page, and tries to submit with fake or no data and get valid validation error. In addition to that, we always test ‘Paid Signup in Production’ with real payment. I’m sure, your place someone has valid card to make payment and reimburse or submit credits based on the policy. Best thing is to use real data (payment) to ensure paid premium subscriptions works end to end in production in every major release.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.