Should users be made aware they are beta testing in live environments? Should they be given special instructions? Should they have options to test or not test in "live/production" environments?

  • 1
    What does this question have to do with automated testing? Also, this question is far too broad - what industry are you looking at? Have these users signed up to a beta program? Are they being given access by a feature flag? A special URL? As worded, the answer to this question is "it depends" and then a long list of possible things it can depend on including industry, regulations, and your company's normal practices even before considering any technical considerations. Please read sqa.stackexchange.com/help/how-to-ask then edit your question accordingly. – Kate Paulk Aug 6 '18 at 18:19

Normally beta-test users are aware of they are beta-testers because in beta-testing there is an emphasis on word testing.

Either the beta-testers are obliged to find and report defects depends on whether the beta testing is performed in open or closed model.

In closed model, normally, the professional testers are involved so that they are to find and report defects.

In open beta, normally regular users are involved. They are notified about the software currently is in beta phase hence the quality is not guaranteed. It is up to the users either to report found issues or not.

| improve this answer | |

As pointed out by Kate: "it depends". There are many factors to consider, such as your industry, your product, your users, … moreover, as already said by Alexey, I think the word "beta" is important here because beta users are usually aware that they are using a beta version.

However, since it hasn't been mentioned so far: There is also canary testing:

In software testing, a canary is a push of programming code changes to a small group of end users who are unaware that they are receiving new code. Because the canary is only distributed to a small number of users, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.

Additional resources for canary releases/testing:

| improve this answer | |

Can they do serious harm? If yes, maybe it is wise to tell them.

Should they have options to test or not test in "live/production" environments?

Should they? This greatly depends on the risks. I prefer testing in production with feature toggles. Limiting the number of people that can "test" the new features.

We live in a continuous delivery world. Having separate beta testing and acceptance cycles will just slow you down. Do keep the risk in mind, don't play with peoples lives.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.