0

Currently the requirement we have is to create defects automatically in JIRA when a test automation script fails.

However we are looking for approaches to avoid raising defects if it's an Environment issue or Automation Script/Framework issue.

Here are the approaches which we are thinking of:

  1. For Flaky Tests we will maintain a flag to not raise defects if those are failed
  2. We will raise defects only on failure of Assertion and not on any other type of failure
  3. Will raise defect depending upon presence of some words in the failure log trace/stack trace

While we understand no single approach or combination of approaches would be 100% accurate, it would be a great help if anyone has already achieved some level of accuracy with one or more approaches and can share their approach.

1 Answer 1

3

I have not found value in automating ticket creation from test failures for many reasons such as false failures, flakies, multiple failures for same base reason, etc. The problem I see is that you get too much signal over noise and this leads to low overall quality.

My approach is as follows:

Take the flakes seriously. Also imagine an actual failure in them - you would not want a 'flaky flag' to hide a real failure. My approaches with flakies are:

  • see if the dependent service can be mocked or stubbed
  • see if the test can be re-written to catch the specific error condition without raising a general error
  • remove the test. This is often the best route. I've seen flakies go for months and the whole time they made the entire test not trusted. One quick decision to remove one flaky test can (and has in my experience) completely address this.

As for defects based on some words on some log: I would not travel down that path. Instead focus on making sure that you have good unit tests that test for negative conditions and then good integration and e2e tests

3
  • 1
    I agree with this answer. The other advantage of reviewing the results manually allows you to create the appropriate items in your defect tracking system. A true positive failure means you put in a defect against the system under test. A false positive failure lets you put in an item of the appropriate type for whoever maintains the test. If there's a flaky test, the test team can decide what is appropriate with respect to tracking work and disabling the test. If you have so many test failures that you can't effectively manually review them quickly, that's a whole different beast of a problem.
    – Thomas Owens
    Jan 13, 2023 at 13:50
  • 2
    @ThomasOwens yeah good points. A slew of automatically generated tickets with the title 'element not found' is not good. Jan 13, 2023 at 14:23
  • 1
    Yep. There's absolutely nothing like walking into 50 tickets because 50 tests failed for the exact same problem on a login page or something equally universal. Human triage goes a long way to having clear, actionable defect reports.
    – Thomas Owens
    Jan 13, 2023 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.