How common is testing based on log files analysis?

Question

Here is a document that outlines challenges and solutions of testing of Mars Science Laboratory:

http://compass.informatik.rwth-aachen.de/ws-slides/havelund.pdf

I found the approach of log files analysis unexpected but quite beneficial on many levels. It seems to me that the same approach could be used while testing the more conventional web applications, for example. Does anyone have a practical experience and lessons learned from this way of testing?

Answer 1

Web applications are a little bit different in that the web server deals with multiple, unrelated clients. If you are looking for a pattern, it is not enough to know that page Y followed page X -- you also need to know whether those events occurred in the same session. Sometimes this information is available in the logs; sometimes, not. Of course in a NAT'ed world you cannot necessarily rely on the client's IP address.

Log analysis and automata reverse-engineering are also useful for load-testing, except that instead of analyzing logged events, you generate those events in proportions and patterns that resemble what you see in the logs.

Finally, if you want to reverse-engineer an application's behavior from its outputs, you can better be confident that the application is mostly working.

Answer 2

Yes, we use log monitoring as part of our testing process. Each manual tester is asked to monitor the logs during test (tail -f), to look for hidden errors/exceptions during testing.

Also, our UI will pop-up a dialog if a client side exception is thrown. These usually cause automated tests to fail, and starts the investigation.

Finally, we have a script that runs nightly to aggregate all of the errors & exceptions from the log files, compare to a list of known issues, and send an email to the team for new errors that appear in the logs. This triggers investigation. (same script runs in production as well)

Answer 3

We rely extensively on log analysis, but not for Web applications but for testing embedded code (from low level drivers to applications). As the presentation says one of the first decisions you'll have to take is online vs. offline , this changes your strategy completely. Our Perl framework has a simplified API for analyzing offline logs, and/or handling of synchronous/asynchronous online events. Note that using logs makes it easier to separate messages of the presentation and the logic layers.

Answer 4

Our logs are one of the main objects for analysis, because of applications don't have UI and we get all information from logs and network output. Also, we have special requirements for logging functionality. Usually we set maximum verbosity level and use tools like grep, tail and awk to process log files.