Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Here is a document that outlines challenges and solutions of testing of Mars Science Laboratory:

http://compass.informatik.rwth-aachen.de/ws-slides/havelund.pdf

I found the approach of log files analysis unexpected but quite beneficial on many levels. It seems to me that the same approach could be used while testing the more conventional web applications, for example. Does anyone have a practical experience and lessons learned from this way of testing?

share|improve this question
add comment

4 Answers

up vote 1 down vote accepted

Web applications are a little bit different in that the web server deals with multiple, unrelated clients. If you are looking for a pattern, it is not enough to know that page Y followed page X -- you also need to know whether those events occurred in the same session. Sometimes this information is available in the logs; sometimes, not. Of course in a NAT'ed world you cannot necessarily rely on the client's IP address.

Log analysis and automata reverse-engineering are also useful for load-testing, except that instead of analyzing logged events, you generate those events in proportions and patterns that resemble what you see in the logs.

Finally, if you want to reverse-engineer an application's behavior from its outputs, you can better be confident that the application is mostly working.

share|improve this answer
add comment

Our logs are one of the main objects for analysis, because of applications don't have UI and we get all information from logs and network output. Also, we have special requirements for logging functionality. Usually we set maximum verbosity level and use tools like grep, tail and awk to process log files.

share|improve this answer
add comment

We rely extensively on log analysis, but not for Web applications but for testing embedded code (from low level drivers to applications). As the presentation says one of the first decisions you'll have to take is online vs. offline , this changes your strategy completely. Our Perl framework has a simplified API for analyzing offline logs, and/or handling of synchronous/asynchronous online events. Note that using logs makes it easier to separate messages of the presentation and the logic layers.

share|improve this answer
add comment

Yes, we use log monitoring as part of our testing process. Each manual tester is asked to monitor the logs during test (tail -f), to look for hidden errors/exceptions during testing.

Also, our UI will pop-up a dialog if a client side exception is thrown. These usually cause automated tests to fail, and starts the investigation.

Finally, we have a script that runs nightly to aggregate all of the errors & exceptions from the log files, compare to a list of known issues, and send an email to the team for new errors that appear in the logs. This triggers investigation. (same script runs in production as well)

share|improve this answer
1  
The interesting thing about the Mar Science Laboratory's approach is that it uses patterns across event types to detect errors, as opposed to looking for stack traces or error messages. It appears that they can also use their tool to auto-generate candidate patterns. –  user246 Aug 6 '12 at 19:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.