3

The Therac-25 incident is a well known issue where the users of the product became far more proficient with it than the testers were. This caused a race condition which uncovered a defect. Historically, I have found that timing matters when testing. How quickly, or slowly, you click something or submit data can uncover different issues.

Does anyone have any experience setting up a framework that will test how the system handles different levels of waits between different actions?

To give an example of a scenario:

An application I was load testing passed all tests under a heavy load. While reviewing the code, I noticed a flaw in the design of the system where if there was a very small, consistent load than it would use an immense amount of resources and run constantly against a very small amount of work. I set up a load test that simulated the conditions and verified my findings and sent it back to Development. While this was a performance issue and a performance test, this type of scenario could potentially uncover a functional issue.

2
  • I oh-so-know what I'm tempted to put as an answer. :-))))
    – corsiKa
    Apr 14, 2016 at 21:39
  • Just trying to broaden my knowledge. I think your answer would be suitable for this or an Explicit Wait within an EventDrivenWebDriver with a variable time.
    – Paul Muir
    Apr 15, 2016 at 11:15

2 Answers 2

1

Because of jQuery and Ajax, it is a challenge to get Selenium to wait for everything to load. Testers like repeatable because you need to telll developers how to reproduce the problem. That being said, I have put randomness into my tests.

  javascript{Math.floor(Y*Math.random()+X)}

This will generate an integer in the range [X, (X + Y - 1)]. You could use this to select different drop-down list items, check different boxes, etc. I suppose one could use it to generate variable waits.

I will say that the race conditions that I have found were because I let Selenium "run as fast as it can" and not by introducing waits.

2
  • Well, "run as fast as it can" would be a level of a wait as well, I would think.
    – Paul Muir
    Apr 15, 2016 at 10:24
  • @PaulMuir Implicit waits, yes. But without them the test fails, removing the opportunity to discover race conditions. Apr 15, 2016 at 14:30
0

First: Make your user to be your tester. I.e. release the product with an agent that does the testing. Then, even if users click faster than testers, you will not miss on the analysis.

Second: Make the analysis from the agent to notify your console in real time.

Third: When the agent performs the analysis in real time and completely automatically, you never have to worry about a programmer being able to analyze the issue correctly.

Forth, If the analysis is captured alone with the dynamic information of the system, you will never have to worry about being able to reproduce the issue.

Fifth: 100 agents are less expensive and more reliable that 100 human testers. So if your console receives aggregated results from 100 (or more) agents, your Time-To-Market component affected by your testing just got 100 (or more) times shorter.

And Six: Your testers still win. If the analysis by the agent is explained on the level of source code, your "Black Box Testers" become "White Box Testers" overnight.

Ps: This is not just an opinion, but a personal experience afforded by the tools built by TinkingSoftware.com

Disclaimer: I would not be able to provide such detail answer if I was not involved in building and using this tool, namely "Race Catcher", and this technology "ARM-CM" standing for Application Reliability Monitoring via Collaborating Machines.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.