I have to test embedded software which runs on qemu virtual machine or on real device and can write its output to stdout. So I can check the state of software only by parsing stdout. To communicate with the device I can use Ethernet, so I can open connection and send data or commands. To sum up: I need to test some user scenarios and check the result through stdout parsing

I also need my tool to integrate with Jenkins and produce junit-compatible xml reports.

More Information:

I have experience of Functional/Black Box test automation using C#, and I need to establish a similar type of automation in Python (or some other Linux compatible language) from scratch.

I am looking for automation tools on Python which could be easily integrated into Jenkins workflow, at least they should be able to produce junit-compatible xml reports.

At a first glance, it seems that even unittest library build in Python is enough. It has test cases and test suites, asserts and reporting. But it supposed to be used for unittesting, so maybe I will face some limitations in the future. And same thoughts about other libraries like py.test and nose.

I've looked for more high-level framework and found robotframework. But for me, the syntax looks really strange and mostly suitable for web/gui testing, while I have a software without GUI at all.

Lettuce framework seems pretty much the same, but also warns that junit reports support is not tested yet.

  • 2
    Most frameworks can be integrated into Jenkins. What type of application are you testing? You say it doesn't have a GUI. Can you give more of a background of the application you hope to test and what type of testing you'd like to perform? Mar 21, 2017 at 16:11
  • What are your requirements? What are you trying to test in more detail? Mar 21, 2017 at 21:05
  • @ChrisKenst I have an embedded software which runs on qemu vitual machine or on real device and can write its output to stdout. So I can check the state of software only by parsing stdout. To communicate with the device I can use Ethernet, so I can open connection and send data or commands. To sum up: I need to test some user scenarios and check the result through stdout parsing
    – Semant1ka
    Mar 23, 2017 at 13:49

2 Answers 2


You are right robotframework is very suitable for web/GUI testing, yet - it's hardly limited to that. True that the majority of the users (myself including) utilize it mainly for that - and that's because it is very easy to start with, and has a marvelous layer (library) above the regular Selenium.

Yet, it's a "framework" (obvious by its name :) - it gives you structure, execution and logging, baked-in flow control, and standardized extensibility. Nothing more, and nothing less.

And that's the beauty of it - it does not force you to work on GUI projects; on the contrary - you have enough flexibility to do whatever you want. (a guy, in SO I think, was thinking and asking how to control his raspberry pi pins, an overkill to use RF IMHO, yet - a possibility).

I for one, apart from the expected web tests, have separate suites which are testing only json APIs, others - sql, with no graphical part involved. An interface is just an interface, software by definition is data processing - input of data, business rules ran over it, output of data (sometimes "time" is the data :). Apologies for the lyrical deviation from the topic :)

My point is - don't write off robotframework, especially on the premise its heaviest use (and SO questions) is on GUIs. It is highly versatile and powerful; look at the standard libraries (listed on its webpage), and the community ones (in pypy) - this will give you a hint what is users are actually doing with it. There's a standard telnet library btw, if that's your thing with the device you're testing.

Closing thought - I've used unittest, pytest, and they are great - for their intended purpose - unit testing, on the code. Really great, but are far too bare-metal for any practical functional, integration or system testing. That's why there are higher-level frameworks in the first place, right? :)

Disclaimer: I'm not affiliated to robotframework in any way, but a heavy (ab)user, and strongly advocate for it in any format I can (stackexchange included:).

Take my words with a grain of salt, my "emotional connection" to it may very well be one of those "everything looks like a nail when you have a hammer" :). Yet, after an year and something of usage, 2000+ integration and e2e tests, 3 different systems and one for-the-fun-of-it site scraper which BizDev now want to use for business purposes (poor them :D), I couldn't be more happier with my choice of the framework.

  • Thank you for your full answer about robot framework, I will look at it more carefully. I wonder, is it integrated into any CI infrastructure in your company? I mean, how tests runs from it are triggered and verified?
    – Semant1ka
    Mar 31, 2017 at 12:19
  • @Semant1ka it has a rather robust Jenkins integration. I myself have also done some custom (read: python/bash :D) triggers, as its CLI control is very robust, plus has api for attaching the so-called listeners on execution time. Mar 31, 2017 at 13:54

We've been using Robot Framework (RF) for backend testing of the large distributed system. It worked. System-specific fixtures, e.g., to access company API, were written as Python libraries and were executed by RF scripts. RF offered great reporting. They were executed as part of CI. The biggest problem was refactoring of the scripts (IDEA Intellij does not support that), no support for debugging RF tests in IDEA Intellij IDE and abundance of programming anti-patterns promoted by RF, like global variables. For a tester like me who reads, understands and works with code it was constraining. Don't get me wrong here: RF offers great syntax for representing tabular data but in many things that mature programming language have for ages it is very immature. Plus it is too verbose when it comes to doing such a simple thing as variable assignment.

We were able to rewrite same tests in pure Python, using pytest as an execution framework. It does not have such a nice syntax as RF for representing tabular data (as RF has), but addresses all constrains of RF: it uses programming language for tests, so it's easy to refactor and debug your tests. Pytest does not generate HTML reports but it integrates nicely with existing reporing frameworks: we've used Allure with a success. With a proper DSL you can have almost as pretty (leggible) tests as in RF.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.