# Tag Info

Accepted

### If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?

I'm going to take a different approach than statistics (though I think the other response answers your actual question more directly). Any time I've encountered "a problem that only happens some of ...
• 1,189

### If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?

You must perform the same test equal number of times on both the unfixed version and the supposedly fixed version. You have to show that, The test fails once in every N times randomly, on the unfixed ...
Accepted

### How to deal with flaky tests that have intermittent failures?

Here's the general approach we're currently implementing in our team: Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into ...
• 10.5k
Accepted

### How can I run one spec 100 times using the command line?

If you are using zsh shell: repeat 10 { command } On basic shell: for _ in {1..10} do command; done If you can run the tests in parallel, you can use the GNU Parallel: parallel command ::: {1.....
• 10.5k

### If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?

I suppose this answer could help you You need to decide first at what probability you want to "detect" the problem. This is a nice example to why theoretical knowledge is necessary even for testers. ...
• 7,089

### If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?

While I agree with the other answers saying "dig deeper", to answer the actual math question in the title: If the issue occurs completely at random with probability p, then the chance if it occurring ...

### How to deal with flaky tests that have intermittent failures?

Here is a good general purpose article that deals with flaky tests: https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests What we find is that with image-driven ...
• 111

### How do we deal with occasional (but real) application failures

Yes, often. Intermittent bugs are the most annoying to deal with, especially when you don't even know what triggered the problem. Unless the underlying code/infrastructure changes within the five ...
• 3,370

### How to deal with flaky tests that have intermittent failures?

I normally follow this flowchart when dealing with a flaky test in general.
• 9,932

### If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?

A problem has been observed that sometimes, in testing, produces errors. We don’t actually know the probability that it will produce an error on any given test run, because we can only do a finite ...
• 141

### How to deal with flaky tests that have intermittent failures?

I like to setup a re-run strategy, where we rerun any tests that has failed, until it fails three times in a row. If the test is still failing after two times it will be a real failing test and not ...
Accepted

### How do we deal with occasional (but real) application failures

Intermittent problems are hard. They're hard to reproduce and hard to track down. I work on a process with intermittent issues: Report the issue - I make sure to note that the issue is intermittent ...
• 31.5k
Accepted

### What is the difference between explicit and implicit waits in Selenium?

Authoritative answer from Jim Evans, Selenium contributor: never mix implicit and explicit waits - and stick with explicit. Article explains also your other questions. Converting our code to explicit-...
Accepted

### What techniques can I use to determine the reasons for intermittent automated UI test failures?

My personal opinions: Trying to analyse the testing environment closely, identify as many variables that may have effect on testing results as possible Invite fresh eyes to aid you during this ...
• 9,932

### How do we deal with occasional (but real) application failures

Against flaky test the Googlers themselves contend in vain. So we mere mortals do what Michael Durrant and Kate Paulk suggest. See also Heisenbug.

### which one is more subject to common cause failure: analog or software based system?

It depends The longer answer: There is no general answer to this question. Whether a mechanical solution or a software solution is more likely to fail depends on many factors, including but not ...
• 31.5k

### How to deal with flaky tests that have intermittent failures?

Against flaky test the Googlers themselves contend in vain. (And yes, last link is a reference to a famous sci-fi novel).

### How do we deal with occasional (but real) application failures

Intermittent Failures are the bane of our work. It is important to know that they are: Common to many, if not most, organizations Most commonly UI tests that interact with a browser As common as 1 ...
• 25.2k

### How can I run one spec 100 times using the command line?

Create a script in your home directory, e.g. \$ cat run_spec_100_times.sh cd ~/eq/lynx for i in `seq 1 100`; do rspec spec/features/survey_spec.rb done cd ~ Modify ...
• 25.2k
1 vote

### How to deal with flaky tests that have intermittent failures?

From my experience there are 3 steps you should do: Test quarantine. You can read more about it in one of the following blogs: https://martinfowler.com/articles/nonDeterminism.html https://...
• 19
1 vote

### What techniques can I use to determine the reasons for intermittent automated UI test failures?

screenshot on failure. I had a problem where I was actually on a completely different page than I was expecting because of overlapping elements when I clicked a link. A screenshot can give really good ...
• 764

Only top scored, non community-wiki answers of a minimum length are eligible