4

We are using automation testing tool Cypress to perform regression testing. We make 2 releases in a week so automated regression testing is critical for us.

There are some features on production which are experimental i.e these features will be available only to some percentage of users. Eventually these features might be rolled out to 100% of the users or might get dropped completely. These features stay as experimental for 1-4 weeks.

Even though these are experimental features, we have to regression test these features in every subsequent release. And we have not automated these features yet because there is a 30% chance that these might get dropped completely. Hence we regression test these features manually which is time consuming.

Any suggestions on the regression testing approach for these kind of features?

4
  • 2
    I don't see anything special on this case, you test them the way you test everything else. If it takes time and you want to automate some checks, you need to find time for that, get more people on the team etc., but nothing changes the fact it takes time. If some of the features get dropped, you retire the automated checks for it and on we go. – pavelsaman Feb 10 at 9:05
  • What is the exact problem? You explained your context and your current approach, but I don't exactly how it is problematic, how your team is struggling to fulfill its mission. – João Farias Feb 11 at 9:58
  • @pavelsaman - Both people and time cost money so I am trying to figure out a way to optimize costs since a good percentage of these features will be eventually dropped. – vicky99 Feb 12 at 18:55
  • @João Farias - The problem/question here is about the approach that should be taken for such experimental features. We did think of both approaches but we were still confused and we thought this is definitely something others must have also faced and we wanted to know how they solved this. Maybe there is a guideline or a optimized approach that we might be missing out on. – vicky99 Feb 12 at 18:59
4

Although I don't have a complete answer this question is special, the difference from the common "how do I test" questions is the high frequency of changes. Testing everything thoroughly is one approach, but as @vicky99 explained it's not worth investment for a very short lived feature.

There two approaches to go as I see it. One approach that could help, but highly depends on your product, is having a good test infrastructure- environment, test runner, test API, test framework, documentation for those etc. and then try moving testing to the left towards the developers and early in the development stages.

Having a solid test infrastructure means that a developer can add tests relatively quickly and painlessly, those test might be rudimentary but that's better than having none at all. The test automation specialist of the team will support this effort by maintaining the test infrastructure and adding to it new features in parallel to development of new features.

Up to here this sounds like a normal way a team should work but you must adapt it for speed and efficiency, for example by even better communication between developers, putting test development higher in priority or accepting simpler tests first and only later improving them.

The other approach, which might not stand on it's own, is having excellent monitoring and data analysis. Since you already have AB testing you probably have telemetry being sent back and analyzed. You can use this data to test in production and look for bugs, you should probably have an automated system analyzing the data raising alarms and you should make this system smart looking not only for error reports but also for abnormal usage patterns for example. Since you already have basic testing done using the above approach you can be pretty confident that your features are stable and bugs won't be too common.

1

It looks like a question between the tradeoff between the cost/effort/speed of manual testing and automating the feature.

I assume that you have nice commit-stage verification tests. It means that the part of your feature definition of done is covering the feature with unit and integration tests, regardless of whether it would stay in your app, or would be dropped after A/B testing. If that is not the case - consider moving the testing efforts closer to the development phase, shift left, as was already mentioned.

In that case - you can reduce the amount of manual testing by lowering the risks of introducing a broken business logic that might not be verified. Manual testing should only be performed on a high-level depending on which risks you have and try to mitigate.

In case the feature is about to stay - cover it with E2E tests and eliminate the rest of the manual effort.

The other way is to introduce a record and playback automation before the go/no go decision is taken. E.g. using Cypress Recorder

Technology facing, low-level tests should be still in place, but in that case, you could eliminate the manual effort by temporarily verifying the functionality with those tests. Record and playback is a horrible choice for long-running test automation because of maintenance costs and fragility, but for short-lived features (record, test and throw those tests away) that might work well. Later you can cover the feature with reliable tests using Cypress.

1

It is difficult to fixate on common automation strategy for these kind of features from an automation testing services perspective. A better approach would be take them on case by case terms. For example, it will be better to automate a feature that will stay for 4 weeks than the ones that are expected to be discontinued after 1 week.

Another factor to be considered may be the overall cost saved in terms of time spent by automating them over manually testing them. You will have to consider the maintenance cost. Lastly, how likely is the feature expected to stay, then its worth automating it as you will have to eventually do it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.