Failed automated tests: how to distinguish known and newly introduced bugs?

Question

Use case: Fitnesse is used for automated testing of the web site.

SUT (software under test) contains a known bug. Say, we expect that web page contains "Changes saved successfully" string but this string is missing because of the bug. So in Fitnesse, this test case is marked as red.

Suppose, in another test case we expect that web page contains "A user created successfully" string. And it worked just fine until the last test execution. So, now this test case is also marked as red.

So, now we have red light for two test cases: a well known bug and a newly found bug. The problem is they are both marked as red. So when I have a look at test results I can't distinguish which of them are known and new.

Of course, I can compare test history and see the difference between two runnings (with and without a newly created bug).

Or I may not execute a test case with a known bug.

Or I can tune it so that this test case has always been green and change it when the bug is fixed.

But all this is very inconvenient. What I want is to distinguish two kinds of bugs (a well known bug and new bug) so that:

1. By looking at test results, I could easily say: this is a new bug and those are old. For example: No bugs - green, Already known bugs - yellow, New bugs - red.

2. It is easy to change test case when the bug is fixed.

What are the best strategies for acceptance tests, in general, and Fitnesse, in particular?

Answer 1

Your automated tests are currently set up to answer the question "What isn't working as expected?"

Now you want them to answer the question "What isn't working that I don't already know about?"

Your solution is to change the automated tests so that they account for the "known" bugs in one of two ways:

• You could comment out the tests which find every known bug, so that they don't execute
• You could change the tests which find known bugs to temporarily look for the failure condition, rather than the expected condition

Be careful with either of these solutions! They require you to constantly update your tests every time you write a bug report. And you must remember to re-set your tests whenever you want to again ask the question "What isn't working as expected?"

One other trick (which I have used in the past) would be to leave the tests as they are, but save the output log from your automated tests, and compare it to the prior run. That diff would tell you "What is different in today's run as compared to yesterday's run?" If the log is constructed correctly (for example, log only failures, and never log the current date/time or other dynamic data), this could help give you the answer you need.

The difference between yesterday's log and today's log will indicate any new bugs found in today's run, as well as any old bugs that were fixed in today's run.

Answer 2

I have not used Fitnesse, but I have dealt with the same problem. I think there are two related but distinct issues: handling long-term bugs, and identifying new failures.

Handling long-term bugs

If a test fails because of a known bug (an expected failure), and you know the bug will not be fixed any time soon, you may want to take that into account when interpreting your test results. Your options include the following:

• Not running the test until you expect the bug to be fixed. This may be attractive if the test takes a long time to run. You could accomplish this by commenting out the test. Even better, your test framework may allow you a way to specify an exclusion (a "do not run") list.
• Running the test but annotating the expected failures so that you can distinguish them from other failures.
• Running the test but excluding the expected failures from the results.

Identifying new failures

Sometimes you want to identify failures in the latest run that did not occur in the previous run. Jenkins provides a way to do this. Anyone managing a sizable test suite will want to do this.

Answer 3

I am using bug-workarounds to handle those “long-term” bugs in the test automation. My workflow is following:

1. Create a bug in the bug-tracker. For instance its id is BUG10666

2. Create a Boolean constant with bug id in its name: const bool is_ BUG10666_fixed = false; All the constants for bugs should be placed in a separate file.

3. Use the workaround to make the test green

For instance:

The test was:

String actual = “DZis is a Test”;

String expected = “This is a Test”;

Assert.Equals(actual, actual);

After workaround:

String actual = “DZis is a Test”;

String expected = “This is a Test”;

If (is_ BUG10666_fixed == false) // Workaround for #Put bug description here#

{

expected = “DZis is a Test”;

} Assert.Equals(actual, actual);

And the test case will be green. After the developers fix the issue, change the constant value: is_ BUG10666_fixed = true to disable workaround.

After that you can refactor your code to remove the if-statements for workarounds: // If (is_ BUG10666_fixed == false)

And as you have put all the workaround switches to a separate file, then you can always know which workarounds are enabled and which are disabled.

If you have more than 30 failed tests in your suite, then you have definitely missed some new issue because your tests are blocked with known issues.

And you always can enhance this workaround technique, for instance, to have an array of workarounds, so that you could print all the used workarounds in the test report automatically.

I am reporting the 99% passed automation runs and the list of used workarounds. If one of the workaround had happened – then the last test case in my suite become red and prints all the list of used workarounds.

Answer 4

If you are using a custom framework and a custom reporting/logging mechanism its possible to develop some additional logic around 'failed but known tests'. Once your automation scripts find a 'failed test', custom reporting algorithm should doing a look-up from a 'known issues' list. If the failure is present in your known issues list, then your reporting algorithm can flag this issue as a 'known issue' in your logs/reports. The advantage of this mechanism is that you do not have to change your automation code as a workaround. I'm not sure however if the Fitness provides a mechanism to flag failures as 'known issues'

Answer 5

One solution to your dilemma is to export the results into a spreadsheet after a run and use macros/formulae to compare outputs.

1. On the test history page choose file > save as in your browser and choose single page (not whole site)
2. Open that file in excel
3. I usually use text to columns using the dot as the delimiter
4. You can then use the result column to view your results
5. Copy and past into a master list to make comparisons

I used to take it one level further and use a pivot table on another sheet to make my comparison(s). Once you have a pivot table set up, it is easy to change the data it is using. The secondary use of the pivot table is you all of a handy set of data to add to reports!