Regression/Automation testing for experimental(ABtest) features

Question

We are using automation testing tool Cypress to perform regression testing. We make 2 releases in a week so automated regression testing is critical for us.

There are some features on production which are experimental i.e these features will be available only to some percentage of users. Eventually these features might be rolled out to 100% of the users or might get dropped completely. These features stay as experimental for 1-4 weeks.

Even though these are experimental features, we have to regression test these features in every subsequent release. And we have not automated these features yet because there is a 30% chance that these might get dropped completely. Hence we regression test these features manually which is time consuming.

Any suggestions on the regression testing approach for these kind of features?

Answer 1

Although I don't have a complete answer this question is special, the difference from the common "how do I test" questions is the high frequency of changes. Testing everything thoroughly is one approach, but as @vicky99 explained it's not worth investment for a very short lived feature.

There two approaches to go as I see it. One approach that could help, but highly depends on your product, is having a good test infrastructure- environment, test runner, test API, test framework, documentation for those etc. and then try moving testing to the left towards the developers and early in the development stages.

Having a solid test infrastructure means that a developer can add tests relatively quickly and painlessly, those test might be rudimentary but that's better than having none at all. The test automation specialist of the team will support this effort by maintaining the test infrastructure and adding to it new features in parallel to development of new features.

Up to here this sounds like a normal way a team should work but you must adapt it for speed and efficiency, for example by even better communication between developers, putting test development higher in priority or accepting simpler tests first and only later improving them.

The other approach, which might not stand on it's own, is having excellent monitoring and data analysis. Since you already have AB testing you probably have telemetry being sent back and analyzed. You can use this data to test in production and look for bugs, you should probably have an automated system analyzing the data raising alarms and you should make this system smart looking not only for error reports but also for abnormal usage patterns for example. Since you already have basic testing done using the above approach you can be pretty confident that your features are stable and bugs won't be too common.

Answer 2

It looks like a question between the tradeoff between the cost/effort/speed of manual testing and automating the feature.

I assume that you have nice commit-stage verification tests. It means that the part of your feature definition of done is covering the feature with unit and integration tests, regardless of whether it would stay in your app, or would be dropped after A/B testing. If that is not the case - consider moving the testing efforts closer to the development phase, shift left, as was already mentioned.

In that case - you can reduce the amount of manual testing by lowering the risks of introducing a broken business logic that might not be verified. Manual testing should only be performed on a high-level depending on which risks you have and try to mitigate.

In case the feature is about to stay - cover it with E2E tests and eliminate the rest of the manual effort.

The other way is to introduce a record and playback automation before the go/no go decision is taken. E.g. using Cypress Recorder

Technology facing, low-level tests should be still in place, but in that case, you could eliminate the manual effort by temporarily verifying the functionality with those tests. Record and playback is a horrible choice for long-running test automation because of maintenance costs and fragility, but for short-lived features (record, test and throw those tests away) that might work well. Later you can cover the feature with reliable tests using Cypress.

Answer 3

It is difficult to fixate on common automation strategy for these kind of features from an automation testing services perspective. A better approach would be take them on case by case terms. For example, it will be better to automate a feature that will stay for 4 weeks than the ones that are expected to be discontinued after 1 week.

Another factor to be considered may be the overall cost saved in terms of time spent by automating them over manually testing them. You will have to consider the maintenance cost. Lastly, how likely is the feature expected to stay, then its worth automating it as you will have to eventually do it.

Answer 4

Develope a strategy for automation of testing of AB features.

This was an interesting example and I now need to add AB testing to the (long) list I have for reasons why automated tests don't get written.

You currently have a process that encourages tests not being written. It is:

• Write a feature under AB testing and don't write automation for it - hey maybe it won't be used anyway
• Determine which of A and B to use
• Determine that the features works as it was already tested in production during the AB test
• Pick the next feature to work on with AB testing

This results in:

• automated tests are not written
• application code is not written with testing in mind
• application code is hard to test without a lot of additional effort
• product considers the feature done given that a subset of users has tried it out
• product leaders now focus on developing the next feature to be AB tested

To solve this have a meeting and determine what the actual strategy that you want. For example you could write tests during the AB phase. A 70% chance a test will be used is good enough for me - and it will lead to 'testable' code being written - or you could write tests once the AB testing is wrapped up and A or B is chosen.

Basically you need to get folks together, discuss the issue, determine a strategy and document and evangelize it.

This requires a lot of self-empowerment and a suitable environment to make the changes you see fit.