Main observations in unstable scripts are as follows:
- Selenium tests are usually unstable because of the unnecessary complexity created in your test framework
- Enabling parallel execution without keeping in mind that one test may affect the state of the system that in turn affect the other test running in parallel.
- Using deprecated modules
- Incorrect configuration of modules that are used in the framework
- Not using explicit wait for asynchronous actions
- Using tools that are not meant for testing the software under test (e.g., use protractor for angular webpages than selenium)
- Developing test in one OS and running it in another OS, in CI/CD test agents may have different OS and configurations than what is used in the development PC
There are many such reasons. Also, whenever investigating such issue, follow the below steps in the order mentioned:
1. Trust your code and doubt software under test (SUT):
If everything was working fine and the test starts to fail suddenly. Instead of debugging your code for issues, start with checking the actual product. Do some visual inspection and see whether the development team has modified the element or the element is no longer being displayed.
2. Trust your code and doubt the environment:
If everything was working fine locally and failed as soon as you integrated to CI/CD. Then investigate the product behaviour in the test server. Mostly due to OS and configuration differences the product won't work as it would in local (raise a bug)
3. Now doubt your scripts (using absolute XPATH):
You might be using an absolute XPATH. This causes flaky tests when the DOM structure changes. Use a relative XPATH (CSS would be more recommended). Never use xpath/ if you have a unique ID/name to identify an element.
4. Now doubt your scripts (not using explicit wait):
Sometimes scripts lack explicit wait and try to interact with dynamic elements, and this causes the test to fail, because it tries to interact with the element before it is even available in the DOM.
5. Now doubt your scripts (handling spinners):
Sometimes spinner takes time to get displayed. So, if you are just checking for the invisibility condition then it will return true and try to interact with the next element before the actual spinner event completes.
So, first, check the visibility of spinner and then check for invisibility before interacting with other dynamic elements.
6. Now doubt your scripts (not handling iFrames):
Sometimes an element will be inside iframes and scripts won't switch between frames before interacting with these elements.
Check whether any parent element contains the tag frame or iframe to determine whether the element is inside an iframe.
