Return to Answer

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

• Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis of root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
• Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
• There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

• Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis of root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
• Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend to be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
• There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

• Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis or root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
• Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
• There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

• Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis of root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
• Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
• There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine (see Martin Fowler's Eradicating Non-Determinism in Tests), repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results.

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

• Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis or root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
• Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
• There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. However, unstable end-to-end tests are good at finding more bugs, but test results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

• Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis or root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
• Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
• There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.