6

I have a test that is making sure that all elements are visible on a page, my question is should I have 18 separate test cases or just one?

Example, which is better

let baseURL = Cypress.env("BASEURL")
beforeEach(() => {
    cy.visit(baseURL + '/creditApp/index')
});

it('be.visible firstName when user visits credit app', () => {
    cy.get(CreditAppPage.firstNameID).should('be.visible')
});

it('be.visible lastName when user visits credit app', () => {
    cy.get(CreditAppPage.lastNameID).should('be.visible')
});

or

let baseURL = Cypress.env("BASEURL")
beforeEach(() => {
    cy.visit(baseURL + '/creditApp/index')
});

it('be.visible firstName when user visits credit app', () => {
    cy.get(CreditAppPage.firstNameID).should('be.visible')
    cy.get(CreditAppPage.lastNameID).should('be.visible')
});
5

Three things should be considered, all equally important:

  • Performance: Will creating several tests increase the suite execution time? It highly depends on your framework. Even if the time to bootstrap a test is short, it can be damaging on the long-run;
  • Test dependency: If you use only one test, you should be able to make the assertions independent - if the first fails, only the ones that don't make sense anymore should be skipped. Often a assertion failure stops the test. Again, depends on your framework;
  • Maintainability: Do expanding on multiple tests make it difficult to change your tests?

    The following is easier to expand than breaking into multiple tests, e.g.:

    it('be.visible multiple when user visits credit app', () => {
        elements = [CreditAppPage.firstNameID, ..., CreditAppPage.ninethNameID]
        elements.foreach((element) => element.should('be.visible'))
    }); 
    
| improve this answer | |
  • 2
    I'd add that the output is important too. If it's not clear which assertion failed, then the aggregated assertion type testcases can get expensive quickly when they fail. – ernie Nov 30 '17 at 0:36
2

I decide these things by asking a question:

If a responsibility fails one assertion, do I still want information from the other assertions?

Usually the answer is yes, in which case I put the assertions in separate tests.

But for some situations, a failure in one assertion renders some other assertions meaningless or distracting. For those situations, I put the other assertions in the same test.

| improve this answer | |
  • It's also important to differentiate between soft and hard assertions. Hard assertions are implemented this way that even if put in the same test and first of them fails, the next ones won't be evaluated. That's where soft assertions should be put. – dzieciou Apr 1 '18 at 15:57
2

Usually my tests have one assertion per test. However for this specific issue for UI tests there is one single over-riding factor (in this discussion about 1 or many) and it is

Performance

In UI tests I intentionally break that rule for 1 reason - speed.

UI tests, e.g. browser tests are very slow and if they are used to give quick feedback you have to keep them fast, e.g. 1 set of assertions per UI test. This can be the difference between a set of tests taking 1 minute or 30 minutes so the difference is order of magnitude not just minor savings

| improve this answer | |
1

It really is a matter of preference, i personally prefer not to nest all in one test and split across 18 tests simply because of the reporting data.

Once all are nested in one test if one fails the remaining values will not be evaluated, hence you might miss a failure.

| improve this answer | |
  • basically not true, as javascript runs all commands in parallel. Can be a problem in sequential frameworks. – Dee Nov 30 '17 at 11:11
1

Such test case we call a GUI tests or GUI Test cases and those have GUI in their name indicators.

One test, multiple objects in it

Usually, we put all GUI checks on one screen into one test. This is according to multiple test methodologies.

Additionally, In javascript (ES) all checks are fired simultaneously anyway, so you will not miss other checks if one of them will fail. The only disadvantage is, that you will not get one "test result" per object and you should get it from the test error report, for example, the code-line, if something went wrong.

Multiple tests one for each object

If one of the persons, who are checking the automated test log is not skilled enough, you would need to get human readable test reports. Then you can use the test suite name as the name of the page and to have one test per object. But consider a lot of it(...) around each test. data-drivenen approach is then recommended, so for example in JSON object could be the object name, object xpath or class name and the it(...) should be inside of the foreach cycle. The it name could be calculated as "checking if the element "+elementName+" is present". Transfering the objects to array will be necessary for usage in foreach.

Then you can get test results for one-page TestSuite and for each object. This approach is a bit antipattern. In reality, you will not do thousands of test cases, but just a few of them, so it can break your statistics if you monitor such numbers/KPI's.

Resume

Decision is up to you. I personally would choose the methodical approach. One test, multiple objects. in the test suite for the page is then not only the GUI check but also other tests of the functionality, for example, multiple variants of the functional test, etc.

| improve this answer | |
1

I would make 18 different tests. It's better for debugging. If you group everything into one test, when it fails, you won't really know which one failed. And then you will have either rerun the test and watch or read the logs.

Also, it could be that only one element failed to load. By having separate tests, you can narrow down what caused the error easier.

Additionally, you are making all your tests coupled to one another if you group them up. That means if a selector changes, all the tests will fail rather than just one.

| improve this answer | |
0

In my E2E test, there would be single function called 'verifyPageElements' where I would pass the array of element s which supposed to be present on the page.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.