8

The system I'm working on has error logs which show various types of visible (front end user gets an error message) and none visible errors (missing image file etc).

Some of these errors are seemingly unreproducible or only reproducible in a live environment for a specific site (Assuming that the same application is being run on different sites (clients)), which makes it hard to debug.

I track the errors being raised across various sites running the same application (different instances) and look for front end user visible errors or high volumes of the same type of error being raised and put these into the development queue as a bug. I've been getting cases coming back from development as they can't fix them, as they are unreproducible and because I haven't supplied them with enough detail.

So my question is, if you know errors are being thrown consistently but it seems like the effort required to fix and reproduce the issue is too high. Then what should you do?

One of my suggestions has been to make it easier to debug the issue and obtain more information. Then we can try and resolve the issue when we know more.

Improve this question
3
  • Sounds like what happens when you start tracking all JS errors in a live website with window.onerror
    – Petah
    Commented Jan 12, 2015 at 23:33
  • Yes we are. Do you have any alternative suggestions?
    – Vincent Brazil
    Commented Jan 13, 2015 at 11:22
  • Just FYI, a lot of errors stem from browser plugins/extensions, and intermittent connection issues (i.e. a script not loading) and not actual errors in an application.
    – Petah
    Commented Jan 13, 2015 at 21:16

8 Answers 8

Reset to default
7

I would report the issue, but describe clearly that I cannot reproduce it. As a tester it is not my call to decide how much effort is put in reproducing the issue. I think we have signaling role.

Risks

Its nice if the report contains an estimated risk, do real users have problems and is it possible they get stuck and maybe cannot use a certain feature, which lowers the user experience. How many users are effected, how often etc?

Debugging

From a code perspective an issue might be easy understandable, even when its not reproducible. If the developers are also having a hard time with it, they could add extra logging. Now monitor the issue again and see if it can be fixed with the extra information. Repeat until fixed.

Rejection

Even if the bug is rejected as un-reproducible it might be worth the info in the bug-tracker when an actual user complains about the same issue. This way any research will not be lost and can be re-used in a later stage.

Improve this answer
2
  • I almost want to downvote this due to the paid per issue. Overall though great advice. Also CYA applies here.
    – Paul Muir
    Commented Jan 19, 2015 at 17:17
  • @PaulDonny Lol, that was a joke. I removed the "Also I get paid per issue I report ;-)" from the answer. I don't actually get paid per defect.
    – Niels van Reijmersdal
    Commented Jan 19, 2015 at 17:24
4

The problem should be reported, with a note in the report that you can't reproduce it consistently (or at all) and that you will update the report as more information is available.

Your next steps should include some mix of the following strategies:

  • If you can increase the level of logging, do so and leave the logging in place until you get a repeat of the error. Sometimes this is something you can send to a customer who has the most problems with the error, and have them send the log once the error recurs. It's possible the sequence of events recorded by the log will be enough for the developers to work out what's happening (you may not be able to verify these and have to rely on customers reporting that the problem no longer occurs).
  • If you can get screen recordings or keylogging, do that.
  • If there's a stack dump of the error, attach it with whatever was logged previously.
  • If it's a web application and you can extract the web server logs, do so. Digging through server logs can be painful, but it can also give you critical information. If there isn't enough information in the server logs, see if you can get the logging level increased until the problem recurs and you can capture that information.
  • You may need to capture logs from multiple sources to trace the problem. I had a case recently where the problem was ultimately users re-trying a job before the original job had completed and in the process corrupting the data the job was using. To trace it I needed logs and timestamps from multiple services in order to recreate what actually happened.
  • For web applications, you may also need to capture the client information string and recreate the user environment as closely as possible in a virtual machine
  • Once you work out a more or less reliable way to reproduce (if you do), update the report with that information. You may need to intermittently try to reproduce for weeks - or months.

Even if the problem report sits for months in a not-reproducible state, it's there as a reminder that the problem exists, which means if it starts happening more frequently the problem can be re-prioritized.

Improve this answer
3
  • This is thorough and seems like a good course of action. However I don't think all bugs merit this level of effort (if you assume that an application error which isn't visible to the user, is less important). In my case I'm looking at the error logs of a web application and some of the errors aren't visible to the end using but they are still clogging up the logs and would ideally be rectified. I realise it seems circular to ask, how do you deal with these situations and i'm essentially suggesting that they aren't always worth a thorough investigation.
    – Vincent Brazil
    Commented Jan 12, 2015 at 13:12
  • 1
    @VincentBrazil If they clog the log then maybe they should be removed from the loggin cycle. I remember a time we stopped looking at the log files, since it was just to much garbage to go through. Ignoring the full log eventually might be worse then spending some time on these issues.
    – Niels van Reijmersdal
    Commented Jan 12, 2015 at 13:35
  • @VincentBrazil - if you're certain that these errors have no impact on the application, then yes, you can choose not to log them. There are errors that fall into this category: you just need to monitor for a while to be sure that there is no impact (sometimes hidden errors can put the application into an unstable state, which leads to noticeable errors elsewhere).
    – Kate Paulk
    Commented Jan 13, 2015 at 11:47
3

Raise it anyway and give as much detail as possible (time, date, error log, test evidence, etc.). If it affects something further down the line, this will cover your back if a project manager or stakeholder asks why something wasn't done about it.

I've experienced that a few times, where something's not reproducible so the developer just says "I can't fix it if you can't reproduce it". Our job is to raise defects against requirements - it's not your issue if developers push back and decide not to follow it up :)

Improve this answer
3

Report the bug. The developer(s) might not be willing or able to address the problem for one reason or another, but the developer(s) can't address the problem if it goes unreported.

Give as much detail as you possibly can. I remember a bug I saw a few months ago which only occurred because the user was running Internet Explorer as a specific account in a production environment off the company VPN. If any of those four variables changed, the bug didn't show up. The developers had trouble reproducing the bug because, naturally, their machines were on VPN, so even when trying different browsers, different accounts, and test/staging/production, the bug failed to appear.

The developer(s) might sigh or go bug-eyed if you are overly verbose, but the developer(s) will actively complain and get no bugfixing done if you don't give them enough.

Improve this answer
3

Globally, you want to determine the risk and take appropriate action (which may be no action). So you really want to know what the effect of the bug is, and how often and/or likely it occurs. If its rare and results on some minor UI anomaly, its not worth pursuing. If it happens every time on your landing page, or you're losing customer money, then at a minimum you'll want to add additional diagnostic code, and probably want to explore other ways to quickly nail down and fix whatever is causing this problem. More typically, the risk is somewhere in between, so the level of effort to address it is a judgement call.

I started with the word 'globally' because in reality, the ability and responsibility to do all that is generally split between the development team, the test team, and possibly many other stakeholders. So in addition to the technical answer, there may also be some bureaucratic problem-solving. Keep the global goal in mind, and hopefully you or someone on your team has skills in that area.

Improve this answer
1

Apart from what others have suggested, you can also try 'Session Replay' tools. You get to see the exact actions performed by the users at the time when errors were logged in the backend. This has a great effect on developers. When they see this, they instantly believe that something is wrong and drive it to closure.
I don't know if there are any Opensource tools in this category but we use the one from Splunk. My last project used a similar tool from Adobe. Splunk

Improve this answer
0

if you see an issue it must be reported even if it is not easy to reproduce it. because it may make a bigger issue according it and it may be easily solved so you must discuss it with the developer and for the case that it happened for one time or not frequently i suggest to use video recorder like Screencastify on Google chrome browser.

Improve this answer
1
  • Not all errors are generated by the tester and therefore can't be recorded. Developers push back on issues they can't reproduce or resolve easily, which means you end up in a stalemate where the tester is reporting an issue and a developer is saying it can't be fixed. This isn't an effective use of anyone's time.
    – Vincent Brazil
    Commented Jan 12, 2015 at 10:15
0

As a first step, I would recommend doing an analysis of the source code to determine which are the sources of non-determinism of the application (i.e. anything that has the potential to make the application execute differently). Generally there are two types (with a lot of subtypes each):

-input;

-memory interleaving

you can find more details about sources of non-determinism here.

Boring-but-'doable' possibility: If your application only has only sources of the first type (input), it may not be so hard to achieve deterministic replay: log all types of input, using something like aspect-oriented programming, program instrumentation, or other logging tools. Once the error is induced, you can use the log to reproduce the bug deterministically.

(Very) Hard-to-address possibility: If memory interleaving has direct influence in the problem (e.g. multiple threads accessing the same resource, with poor or no synchronization), logging is not your friend. This is because logging the non-determinism of thread accesses causes severe performance degradation (you may end up having to reboot your system) and very large logs (too large). In this type of situation, I suggest you spend some time looking at the implementation of thread synchronization in the application. Otherwise, you may also take a look at the most recent research developments in this area here (to the extent of my knowledge this is the most recent), they provide C++ and Java implementations.

Hope this helps.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.