5

I am using Cypress for end-to-end testing on a Web App, and for some browsers I noticed slight changes in the output when changing the build agent.

It is something that I didn't even consider before, but now I am wondering if it makes sense to do end-to-end testing on multiple build agents (or operating systems), or if it is enough to do it just on a particular build agent?

Improve this question
3
  • "slight changes"? please elaborate and gives details and examples. Also consider making details and title match more closely.
    – Michael Durrant
    Commented May 7, 2020 at 12:28
  • It's difficult to track because we're constantly updating the Web App, the tests and even the pipelines, but when I'll have a concrete example, I will update my answer.
    – ccoutinho
    Commented May 7, 2020 at 13:04
  • What I meant with slight changes is that the output was not exactly the same. The final result of the tests was indeed the same, but by looking closely to the whole output when looking for the causes of an error, I noticed some differences in the outputs of our tests in Firefox between running in ubuntu and in windows. These changes were irrelevant in that case, but should I be prepared for cases in which they will be relevant?
    – ccoutinho
    Commented May 7, 2020 at 13:04

3 Answers 3

Reset to default
2

Discover the actual/prospect usage matrix to drive testing based on risk:(Sample Matrix)

enter image description here

Testing is purely risk driven activity. I would suggest to find out the production usage base matrix (OS/Browser/devices wise) to find out the actual risk to drive the test efforts accordingly.

It only makes sense to test on wider combinations of configurations if the actual/future prospective usage is expected to be on that scale otherwise it should be tested functionally only on major OS/Browser/Device combinations.

Improve this answer
2
  • Wow that's an excellent template
    – PDHide
    Commented May 9, 2020 at 16:21
  • Thanks for the comment!
    – Vishal Aggarwal
    Commented May 9, 2020 at 23:41
3

Depends on the goal of the test.

I like to differentiate test goals, for example between:

  • Pure Functional, Tests which check if the functionality works in general. This should work on a stable pipeline, simple and fast for quick feedback loops.
  • Browser/OS/Device compatibility, tests to garantuee it works on multiple different devices.

Now you need to figure out where the risks are and how many of whichs tests you want to automate and run on each change.

But I would want the results of my stable, so not to many different agents and certianly not random agents with different configurations. Any changes should be thought out and chosen for the specific goal. Or else you will keep getting random flaky tests that you need to analyze.

Improve this answer
2

It depends on the context.

If your users are using different browsers and operating systems, or desktop/mobile versions of your app, or different versions of browsers, then it is a good idea to test on multiple build agents.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.