Timeline for Why are Selenium tests so unstable?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 13, 2021 at 9:48 | comment | added | PDHide | @BenjaminGruenbaum Yes but good logs , reports, screenshot , helps in pinpointing what went wrong without wasting time in debugging. If something is failing because of order of test executed then its not a flaky test its an observation and we need to triage properly what order and why and is it supposed to happen . | |
| Oct 31, 2020 at 11:44 | comment | added | PDHide | @peter dictionary.cambridge.org/dictionary/english/doubt | |
| S Oct 31, 2020 at 10:11 | history | suggested | Peter Mortensen | CC BY-SA 4.0 |
Copy edited (e.g. ref. <https://www.youtube.com/watch?v=uU-RbEEolw0&t=9s>). Used more standard formatting (we have italics and bold on this platform).
|
| Oct 31, 2020 at 2:27 | comment | added | Peter Mortensen | What do you mean by "doubt your scripts"? Do you mean "doubt" in the Indian English sense? | |
| Oct 31, 2020 at 2:26 | review | Suggested edits | |||
| S Oct 31, 2020 at 10:11 | |||||
| Oct 29, 2020 at 19:44 | comment | added | Benjamin Gruenbaum | @PDHide I mostly agree - but I think making tests stable once you get to rather large scale - can get very tricky. I've seen a lot of E2E tests by big companies (Microsoft/Salesforce/Autodesk/Wix etc) and some breakage is surprising (Testim noramalizes all of this..). Some examples: updating the React/Angular version, updating libraries, new Chrome version, new ChromeDriver/SafariDriver/GeckoDriver version, grid xvfb version updates, OS differences within linuxes and x11 that cause input_injector_x11 behave slightly differently, a/b testing tools and SDKs and of course - new developers :] | |
| Oct 29, 2020 at 18:39 | comment | added | OpenAI was the last straw | Hopefully, if the you're even remotely Agile and developers have made an intentional change to something with automated tests, you'll be kept "in the loop" and know to expect the failure in advance, in which case you can skip some of these steps. Though if your development team still follows Waterfall (aka "Keep the designs secret from the developers and the development secret from the testers"), this may not be the case. | |
| Oct 29, 2020 at 16:17 | comment | added | PDHide | @Benjamin Yes I was just pointing out that , in my experience flaky test are mostly as a result of bad framework design . If a developer could work on making the product stable it won't be that difficult to make a test framework stable . | |
| Oct 29, 2020 at 15:46 | comment | added | Benjamin Gruenbaum | @PDHide the point I was trying to make (I agree with your answer which is why I upvoted it - and was replying to the first comment and not your post) is that even if it sounds easy to write "non-poorly written tests", once you have a sufficient number of them even a small percent breaking can be terrible. | |
| Oct 29, 2020 at 15:02 | comment | added | PDHide | @BenjaminGruenbaum there was no mention that there where 1000s of tests, it mentions few scripts | |
| Oct 29, 2020 at 13:52 | comment | added | Mate Mrše | @BenjaminGruenbaum I agree. That was a joke, btw. | |
| Oct 29, 2020 at 13:35 | comment | added | Benjamin Gruenbaum | @MateMrše tests are like regular software - it's not good enough to call it "poorly written". When you have 1000s of tests just one percent breaking can be a lot of work... | |
| Oct 29, 2020 at 12:31 | comment | added | Moorpheus | Great answer. As an extension to point 3, I'd also say - if you don't have unique ids/names then ask for them! It's often trivial for developers to add this in but the benefit to you for automation is huge. | |
| Oct 29, 2020 at 7:56 | comment | added | Mate Mrše | I was going to say "your tests are unstable because they are poorly written", but this answer is so much better. | |
| Oct 28, 2020 at 19:04 | history | answered | PDHide | CC BY-SA 4.0 |