0

I am having a hard time finding any test frameworks or plugins for frameworks that support saving "validation data strings" as part of the test results. I find this a little surprising so I am hoping that I am just missing something. Essentially, I'd like to be able to make a call of some sort that would store debug, performance, or other data along with the test case (test method) results.

This may deviate from standard xunit but to me seems valuable and I was wondering if I need to somehow add this feature myself to an existing framework or make my own lightweight framework that supports this.

Any input much appreciated. Even a pointer to a non python framework that supports this would be interesting.

  • Do you mean logging frameworks or plugins? I would have thought any kind of logging plugin would be able to work with your testing framework. I'm not familiar with Python, but I'm doing plenty of custom logging with the testing I work with (mostly Microsoft CodedUI and load testing). – Kate Paulk Feb 6 '18 at 16:36
  • Hi Kate, thanks for your comment. I was thinking more along the lines of a way to log that is integrated with the test results in the end. In other words, if I have a test method that logs into a home page, for instance, I could log the time it takes the page to load after clicking the login button and have that log message inserted into the test results. I am thinking that I may have to write this code myself but I wanted to be sure that I wasn't missing something already out there. – chrismead Feb 6 '18 at 16:58
  • I usually handle that by using the logging framework the test framework supplies. That automatically includes the messages as part of the results. – Kate Paulk Feb 6 '18 at 17:07
  • I guess your results end up in a trx file? Those seem to have more than xunit: github.com/x97mdr/pickles/blob/master/src/Pickles/Pickles.Test/… – chrismead Feb 6 '18 at 19:25
  • Yes, they do. Some of the other frameworks I've worked with allow something similar – Kate Paulk Feb 6 '18 at 20:36
1

When i've written frameworks in the past i would have considered that information loggable and would have logged it to a file.

If you include a lot of information in your xunit file it can expose the limitations of XML parsers , this will slow down your CI system when displaying results and generally make the files hard to deal with.

If you can use the nose test runner there are some options for test output that will help if you use its xunit output.

    -d, --detailed-errors, --failure-detail

Add detail to error output by attempting to evaluate failed asserts 

https://nose.readthedocs.io/en/latest/usage.html

  • That is a good point about the xunit file. XUnit doesn't seem to cover non-failure info and extending it might not give good results. reflex.gforge.inria.fr/xunit.html#xunitReport – chrismead Feb 6 '18 at 19:20
  • I guess I will create an intermediate results format with more info and an xunit exporter. Thanks for your help – chrismead Feb 7 '18 at 20:57
  • At the simplistic level you can just pipe stdout to a file – Amias Feb 7 '18 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.