Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Here is a document that outlines challenges and solutions of testing of Mars Science Laboratory:

http://compass.informatik.rwth-aachen.de/ws-slides/havelund.pdf

I found the approach of log files analysis unexpected but quite beneficial on many levels. It seems to me that the same approach could be used while testing the more conventional web applications, for example. Does anyone have a practical experience and lessons learned from this way of testing?

share|improve this question
    
This isn't really testing, let alone acceptance-testing; log analysis like this is more about maintenance and detecting problems after deployment than testing pre-deployment to ensure the code is correct. –  Yamikuronue Jun 2 at 13:01
add comment

5 Answers 5

up vote 1 down vote accepted

Web applications are a little bit different in that the web server deals with multiple, unrelated clients. If you are looking for a pattern, it is not enough to know that page Y followed page X -- you also need to know whether those events occurred in the same session. Sometimes this information is available in the logs; sometimes, not. Of course in a NAT'ed world you cannot necessarily rely on the client's IP address.

Log analysis and automata reverse-engineering are also useful for load-testing, except that instead of analyzing logged events, you generate those events in proportions and patterns that resemble what you see in the logs.

Finally, if you want to reverse-engineer an application's behavior from its outputs, you can better be confident that the application is mostly working.

share|improve this answer
add comment

Yes, we use log monitoring as part of our testing process. Each manual tester is asked to monitor the logs during test (tail -f), to look for hidden errors/exceptions during testing.

Also, our UI will pop-up a dialog if a client side exception is thrown. These usually cause automated tests to fail, and starts the investigation.

Finally, we have a script that runs nightly to aggregate all of the errors & exceptions from the log files, compare to a list of known issues, and send an email to the team for new errors that appear in the logs. This triggers investigation. (same script runs in production as well)

share|improve this answer
1  
The interesting thing about the Mar Science Laboratory's approach is that it uses patterns across event types to detect errors, as opposed to looking for stack traces or error messages. It appears that they can also use their tool to auto-generate candidate patterns. –  user246 Aug 6 '12 at 19:39
add comment

We rely extensively on log analysis, but not for Web applications but for testing embedded code (from low level drivers to applications). As the presentation says one of the first decisions you'll have to take is online vs. offline , this changes your strategy completely. Our Perl framework has a simplified API for analyzing offline logs, and/or handling of synchronous/asynchronous online events. Note that using logs makes it easier to separate messages of the presentation and the logic layers.

share|improve this answer
add comment

Our logs are one of the main objects for analysis, because of applications don't have UI and we get all information from logs and network output. Also, we have special requirements for logging functionality. Usually we set maximum verbosity level and use tools like grep, tail and awk to process log files.

share|improve this answer
add comment

Logs are pretty much useful in below cases

  • Based on Different levels (Debug, Info) you can find intermediate values, final values and compare against entries in log
  • When there is an Error @ UI level you can capture the step / error details from logs
  • After completing your testing, cursory check on logs for exception would throw any memory leak / out of bounds exceptions, timeouts
  • Because of sprint based cycles not every information is captured in design docs, functional flow, assignments, data comparison, clean up will be much more detailed only from logs
  • mtail, Agent check, textpad, notepad++ are some tools which can quickly parse through logs for exceptions

Logs are useful for QA to understand the technical implementation, identify functional state changes, find exceptions

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.