Functional/Black Box tests automation library on Python

Question

Summary:

I have to test embedded software which runs on qemu virtual machine or on real device and can write its output to stdout. So I can check the state of software only by parsing stdout. To communicate with the device I can use Ethernet, so I can open connection and send data or commands. To sum up: I need to test some user scenarios and check the result through stdout parsing

I also need my tool to integrate with Jenkins and produce junit-compatible xml reports.

More Information:

I have experience of Functional/Black Box test automation using C#, and I need to establish a similar type of automation in Python (or some other Linux compatible language) from scratch.

I am looking for automation tools on Python which could be easily integrated into Jenkins workflow, at least they should be able to produce junit-compatible xml reports.

At a first glance, it seems that even unittest library build in Python is enough. It has test cases and test suites, asserts and reporting. But it supposed to be used for unittesting, so maybe I will face some limitations in the future. And same thoughts about other libraries like py.test and nose.

I've looked for more high-level framework and found robotframework. But for me, the syntax looks really strange and mostly suitable for web/gui testing, while I have a software without GUI at all.

Lettuce framework seems pretty much the same, but also warns that junit reports support is not tested yet.

Answer 1

You are right robotframework is very suitable for web/GUI testing, yet - it's hardly limited to that. True that the majority of the users (myself including) utilize it mainly for that - and that's because it is very easy to start with, and has a marvelous layer (library) above the regular Selenium.

Yet, it's a "framework" (obvious by its name :) - it gives you structure, execution and logging, baked-in flow control, and standardized extensibility. Nothing more, and nothing less.

And that's the beauty of it - it does not force you to work on GUI projects; on the contrary - you have enough flexibility to do whatever you want. (a guy, in SO I think, was thinking and asking how to control his raspberry pi pins, an overkill to use RF IMHO, yet - a possibility).

I for one, apart from the expected web tests, have separate suites which are testing only json APIs, others - sql, with no graphical part involved. An interface is just an interface, software by definition is data processing - input of data, business rules ran over it, output of data (sometimes "time" is the data :). Apologies for the lyrical deviation from the topic :)

My point is - don't write off robotframework, especially on the premise its heaviest use (and SO questions) is on GUIs. It is highly versatile and powerful; look at the standard libraries (listed on its webpage), and the community ones (in pypy) - this will give you a hint what is users are actually doing with it. There's a standard telnet library btw, if that's your thing with the device you're testing.

Closing thought - I've used unittest, pytest, and they are great - for their intended purpose - unit testing, on the code. Really great, but are far too bare-metal for any practical functional, integration or system testing. That's why there are higher-level frameworks in the first place, right? :)

Disclaimer: I'm not affiliated to robotframework in any way, but a heavy (ab)user, and strongly advocate for it in any format I can (stackexchange included:).

Take my words with a grain of salt, my "emotional connection" to it may very well be one of those "everything looks like a nail when you have a hammer" :). Yet, after an year and something of usage, 2000+ integration and e2e tests, 3 different systems and one for-the-fun-of-it site scraper which BizDev now want to use for business purposes (poor them :D), I couldn't be more happier with my choice of the framework.

Answer 2

We've been using Robot Framework (RF) for backend testing of the large distributed system. It worked. System-specific fixtures, e.g., to access company API, were written as Python libraries and were executed by RF scripts. RF offered great reporting. They were executed as part of CI. The biggest problem was refactoring of the scripts (IDEA Intellij does not support that), no support for debugging RF tests in IDEA Intellij IDE and abundance of programming anti-patterns promoted by RF, like global variables. For a tester like me who reads, understands and works with code it was constraining. Don't get me wrong here: RF offers great syntax for representing tabular data but in many things that mature programming language have for ages it is very immature. Plus it is too verbose when it comes to doing such a simple thing as variable assignment.

We were able to rewrite same tests in pure Python, using pytest as an execution framework. It does not have such a nice syntax as RF for representing tabular data (as RF has), but addresses all constrains of RF: it uses programming language for tests, so it's easy to refactor and debug your tests. Pytest does not generate HTML reports but it integrates nicely with existing reporing frameworks: we've used Allure with a success. With a proper DSL you can have almost as pretty (leggible) tests as in RF.