36

I'm going to take a different approach than statistics (though I think the other response answers your actual question more directly). Any time I've encountered "a problem that only happens some of the time" as either QA or a support role it's been an investigative exercise about narrowing down why the event happens irregularly or in what situations it ...


20

You must perform the same test equal number of times on both the unfixed version and the supposedly fixed version. You have to show that, The test fails once in every N times randomly, on the unfixed version. The same test passes every time, or at least fails less often, on the fixed version. You have to show that the only difference between 1 and 2 is the ...


10

I suppose this answer could help you You need to decide first at what probability you want to "detect" the problem. This is a nice example to why theoretical knowledge is necessary even for testers. The simplified version: p is the probability for failure, 1/N in our case then the probability for success is 1-p and the probability to have N successful ...


9

While I agree with the other answers saying "dig deeper", to answer the actual math question in the title: If the issue occurs completely at random with probability p, then the chance if it occurring at least once in n trials is 1-(1-p)^n. Setting this to x (your confidence that the issue has been fixed) and solving for n gives you n = log(1-x)/log(1-p) ...


6

If you are using zsh shell: repeat 10 { command } On basic shell: for _ in {1..10} do command; done If you can run the tests in parallel, you can use the GNU Parallel: parallel command ::: {1..1000} Tutorial videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 Text tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html Your ...


4

A problem has been observed that sometimes, in testing, produces errors. We don’t actually know the probability that it will produce an error on any given test run, because we can only do a finite number of tests on the broken system. If 1 represents a test pass, and 0 represents a test fail, we might have a sequence of results from multiple tests that looks ...


3

Authoritative answer from Jim Evans, Selenium contributor: never mix implicit and explicit waits - and stick with explicit. Article explains also your other questions. Converting our code to explicit-waits-only decreased flakiness the test considerably, I suspect for the reasons Jim explains in his answer linked above. Another good read explaining waiting: ...


2

My personal opinions: Trying to analyse the testing environment closely, identify as many variables that may have effect on testing results as possible Invite fresh eyes to aid you during this process of variable identification, but please do this after you have done it so you can compare your results against theirs Trying to have a very controlled testing ...


2

Create a script in your home directory, e.g. $ cat run_spec_100_times.sh cd ~/eq/lynx for i in `seq 1 100`; do rspec spec/features/survey_spec.rb done cd ~ Modify the script to be executable with chmod +x run_spec_100_times.sh The use it with . run_spec_100_times.sh


1

screenshot on failure. I had a problem where I was actually on a completely different page than I was expecting because of overlapping elements when I clicked a link. A screenshot can give really good information as to what is actually happening.


Only top voted, non community-wiki answers of a minimum length are eligible