Want to identify most accurate and effective approach to raise only actual defects automatically in JIRA on test execution failure

Question

Currently the requirement we have is to create defects automatically in JIRA when a test automation script fails.

However we are looking for approaches to avoid raising defects if it's an Environment issue or Automation Script/Framework issue.

Here are the approaches which we are thinking of:

1. For Flaky Tests we will maintain a flag to not raise defects if those are failed
2. We will raise defects only on failure of Assertion and not on any other type of failure
3. Will raise defect depending upon presence of some words in the failure log trace/stack trace

While we understand no single approach or combination of approaches would be 100% accurate, it would be a great help if anyone has already achieved some level of accuracy with one or more approaches and can share their approach.

Answer 1

I have not found value in automating ticket creation from test failures for many reasons such as false failures, flakies, multiple failures for same base reason, etc. The problem I see is that you get too much signal over noise and this leads to low overall quality.

My approach is as follows:

Take the flakes seriously. Also imagine an actual failure in them - you would not want a 'flaky flag' to hide a real failure. My approaches with flakies are:

• see if the dependent service can be mocked or stubbed
• see if the test can be re-written to catch the specific error condition without raising a general error
• remove the test. This is often the best route. I've seen flakies go for months and the whole time they made the entire test not trusted. One quick decision to remove one flaky test can (and has in my experience) completely address this.

As for defects based on some words on some log: I would not travel down that path. Instead focus on making sure that you have good unit tests that test for negative conditions and then good integration and e2e tests