37
votes
Accepted
If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?
I'm going to take a different approach than statistics (though I think the other response answers your actual question more directly). Any time I've encountered "a problem that only happens some of ...
- 1,189
20
votes
If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?
You must perform the same test equal number of times on both the unfixed version and the supposedly fixed version. You have to show that,
The test fails once in every N times randomly, on the unfixed ...
18
votes
Accepted
How to deal with flaky tests that have intermittent failures?
Here's the general approach we're currently implementing in our team:
Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into ...
- 10.5k
12
votes
Accepted
How can I run one spec 100 times using the command line?
If you are using zsh shell:
repeat 10 { command }
On basic shell:
for _ in {1..10} do command; done
If you can run the tests in parallel, you can use the GNU Parallel:
parallel command ::: {1.....
- 10.5k
11
votes
If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?
I suppose this answer could help you
You need to decide first at what probability you want to "detect" the problem.
This is a nice example to why theoretical knowledge is necessary even for testers.
...
- 7,089
9
votes
If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?
While I agree with the other answers saying "dig deeper", to answer the actual math question in the title:
If the issue occurs completely at random with probability p, then the chance if it occurring ...
6
votes
How to deal with flaky tests that have intermittent failures?
Here is a good general purpose article that deals with flaky tests:
https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests
What we find is that with image-driven ...
- 111
5
votes
How do we deal with occasional (but real) application failures
Yes, often. Intermittent bugs are the most annoying to deal with, especially when you don't even know what triggered the problem.
Unless the underlying code/infrastructure changes within the five ...
- 3,370
5
votes
How to deal with flaky tests that have intermittent failures?
I normally follow this flowchart when dealing with a flaky test in general.
- 9,932
4
votes
If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?
A problem has been observed that sometimes, in testing, produces errors. We don’t actually know the probability that it will produce an error on any given test run, because we can only do a finite ...
- 141
3
votes
How to deal with flaky tests that have intermittent failures?
I like to setup a re-run strategy, where we rerun any tests that has failed, until it fails three times in a row. If the test is still failing after two times it will be a real failing test and not ...
- 32.5k
3
votes
Accepted
How do we deal with occasional (but real) application failures
Intermittent problems are hard. They're hard to reproduce and hard to track down.
I work on a process with intermittent issues:
Report the issue - I make sure to note that the issue is intermittent ...
- 31.5k
3
votes
Accepted
What is the difference between explicit and implicit waits in Selenium?
Authoritative answer from Jim Evans, Selenium contributor: never mix implicit and explicit waits - and stick with explicit. Article explains also your other questions.
Converting our code to explicit-...
3
votes
Accepted
What techniques can I use to determine the reasons for intermittent automated UI test failures?
My personal opinions:
Trying to analyse the testing environment closely, identify as many variables that may have effect on testing results as possible
Invite fresh eyes to aid you during this ...
- 9,932
3
votes
How do we deal with occasional (but real) application failures
Against flaky test the Googlers themselves contend in vain.
So we mere mortals do what Michael Durrant and Kate Paulk suggest.
See also Heisenbug.
2
votes
which one is more subject to common cause failure: analog or software based system?
It depends
The longer answer:
There is no general answer to this question. Whether a mechanical solution or a software solution is more likely to fail depends on many factors, including but not ...
- 31.5k
2
votes
How to deal with flaky tests that have intermittent failures?
Against flaky test the Googlers themselves contend in vain.
(And yes, last link is a reference to a famous sci-fi novel).
2
votes
How do we deal with occasional (but real) application failures
Intermittent Failures are the bane of our work.
It is important to know that they are:
Common to many, if not most, organizations
Most commonly UI tests that interact with a browser
As common as 1 ...
- 25.2k
2
votes
How can I run one spec 100 times using the command line?
Create a script in your home directory, e.g.
$ cat run_spec_100_times.sh
cd ~/eq/lynx
for i in `seq 1 100`;
do
rspec spec/features/survey_spec.rb
done
cd ~
Modify ...
- 25.2k
1
vote
How to deal with flaky tests that have intermittent failures?
From my experience there are 3 steps you should do:
Test quarantine. You can read more about it in one of the following blogs:
https://martinfowler.com/articles/nonDeterminism.html
https://...
- 19
1
vote
What techniques can I use to determine the reasons for intermittent automated UI test failures?
screenshot on failure. I had a problem where I was actually on a completely different page than I was expecting because of overlapping elements when I clicked a link. A screenshot can give really good ...
- 764
Only top scored, non community-wiki answers of a minimum length are eligible