5

We are using automation testing tool Cypress to perform regression testing. We make 2 releases in a week so automated regression testing is critical for us.

There are some features on production which are experimental i.e these features will be available only to some percentage of users. Eventually these features might be rolled out to 100% of the users or might get dropped completely. These features stay as experimental for 1-4 weeks.

Even though these are experimental features, we have to regression test these features in every subsequent release. And we have not automated these features yet because there is a 30% chance that these might get dropped completely. Hence we regression test these features manually which is time consuming.

Any suggestions on the regression testing approach for these kind of features?

4
  • 2
    I don't see anything special on this case, you test them the way you test everything else. If it takes time and you want to automate some checks, you need to find time for that, get more people on the team etc., but nothing changes the fact it takes time. If some of the features get dropped, you retire the automated checks for it and on we go.
    – pavelsaman
    Feb 10, 2021 at 9:05
  • What is the exact problem? You explained your context and your current approach, but I don't exactly how it is problematic, how your team is struggling to fulfill its mission. Feb 11, 2021 at 9:58
  • @pavelsaman - Both people and time cost money so I am trying to figure out a way to optimize costs since a good percentage of these features will be eventually dropped.
    – vicky99
    Feb 12, 2021 at 18:55
  • @João Farias - The problem/question here is about the approach that should be taken for such experimental features. We did think of both approaches but we were still confused and we thought this is definitely something others must have also faced and we wanted to know how they solved this. Maybe there is a guideline or a optimized approach that we might be missing out on.
    – vicky99
    Feb 12, 2021 at 18:59

5 Answers 5

5

Although I don't have a complete answer this question is special, the difference from the common "how do I test" questions is the high frequency of changes. Testing everything thoroughly is one approach, but as @vicky99 explained it's not worth investment for a very short lived feature.

There two approaches to go as I see it. One approach that could help, but highly depends on your product, is having a good test infrastructure- environment, test runner, test API, test framework, documentation for those etc. and then try moving testing to the left towards the developers and early in the development stages.

Having a solid test infrastructure means that a developer can add tests relatively quickly and painlessly, those test might be rudimentary but that's better than having none at all. The test automation specialist of the team will support this effort by maintaining the test infrastructure and adding to it new features in parallel to development of new features.

Up to here this sounds like a normal way a team should work but you must adapt it for speed and efficiency, for example by even better communication between developers, putting test development higher in priority or accepting simpler tests first and only later improving them.

The other approach, which might not stand on it's own, is having excellent monitoring and data analysis. Since you already have AB testing you probably have telemetry being sent back and analyzed. You can use this data to test in production and look for bugs, you should probably have an automated system analyzing the data raising alarms and you should make this system smart looking not only for error reports but also for abnormal usage patterns for example. Since you already have basic testing done using the above approach you can be pretty confident that your features are stable and bugs won't be too common.

1

It looks like a question between the tradeoff between the cost/effort/speed of manual testing and automating the feature.

I assume that you have nice commit-stage verification tests. It means that the part of your feature definition of done is covering the feature with unit and integration tests, regardless of whether it would stay in your app, or would be dropped after A/B testing. If that is not the case - consider moving the testing efforts closer to the development phase, shift left, as was already mentioned.

In that case - you can reduce the amount of manual testing by lowering the risks of introducing a broken business logic that might not be verified. Manual testing should only be performed on a high-level depending on which risks you have and try to mitigate.

In case the feature is about to stay - cover it with E2E tests and eliminate the rest of the manual effort.

The other way is to introduce a record and playback automation before the go/no go decision is taken. E.g. using Cypress Recorder

Technology facing, low-level tests should be still in place, but in that case, you could eliminate the manual effort by temporarily verifying the functionality with those tests. Record and playback is a horrible choice for long-running test automation because of maintenance costs and fragility, but for short-lived features (record, test and throw those tests away) that might work well. Later you can cover the feature with reliable tests using Cypress.

1

It is difficult to fixate on common automation strategy for these kind of features from an automation testing services perspective. A better approach would be take them on case by case terms. For example, it will be better to automate a feature that will stay for 4 weeks than the ones that are expected to be discontinued after 1 week.

Another factor to be considered may be the overall cost saved in terms of time spent by automating them over manually testing them. You will have to consider the maintenance cost. Lastly, how likely is the feature expected to stay, then its worth automating it as you will have to eventually do it.

0

Here are a few suggestions for the regression testing approach for experimental features:

  1. Risk-based testing: You could perform regression testing based on the risk involved. The features that have a high chance of impacting the overall functionality of the application could be tested first, while the features that are experimental with a low risk of impact can be tested later or omitted if they get dropped.

  2. Automated regression testing with conditional logic: You could write automated tests for these experimental features, but include conditional logic to skip the tests if the feature is no longer present. This way, you won't have to maintain a separate manual test case for each experimental feature.

  3. Progressive testing: You could start by manually testing the experimental features and gradually automate the tests as they become more stable. This would reduce the manual testing overhead and allow you to focus on automating the tests that have a higher likelihood of staying in production.

  4. Feature toggle management: If the experimental features are being managed through feature toggles, you could leverage the feature toggle management system to control the regression testing of these features. This would make it easier to turn off the testing of a feature if it is dropped before it is rolled out to 100% of users.

Additionally, you could also consider setting up a separate test environment specifically for these experimental features. This environment could have the toggles enabled, so that you could perform regression testing of these features without affecting the main test environment. Furthermore, you could also set up continuous integration and continuous delivery pipelines for these experimental features, which would automatically run regression tests whenever a change is made to these features. This way, you can ensure that these experimental features are being tested as they are being developed, and you can quickly catch any potential issues before they are rolled out to a larger user base.

0

Develope a strategy for automation of testing of AB features.

This was an interesting example and I now need to add AB testing to the (long) list I have for reasons why automated tests don't get written.

You currently have a process that encourages tests not being written. It is:

  • Write a feature under AB testing and don't write automation for it - hey maybe it won't be used anyway
  • Determine which of A and B to use
  • Determine that the features works as it was already tested in production during the AB test
  • Pick the next feature to work on with AB testing

This results in:

  • automated tests are not written
  • application code is not written with testing in mind
  • application code is hard to test without a lot of additional effort
  • product considers the feature done given that a subset of users has tried it out
  • product leaders now focus on developing the next feature to be AB tested

To solve this have a meeting and determine what the actual strategy that you want. For example you could write tests during the AB phase. A 70% chance a test will be used is good enough for me - and it will lead to 'testable' code being written - or you could write tests once the AB testing is wrapped up and A or B is chosen.

Basically you need to get folks together, discuss the issue, determine a strategy and document and evangelize it.

This requires a lot of self-empowerment and a suitable environment to make the changes you see fit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.