Best Practice: BDD and automated GUI testing for Desktop applications

Question

At my company we are developing a Desktop application and regression after regression we finally decided to implement the acceptance tests in addition to the unit tests.

I am relying on this project to use selenium in our application (I tested it and works flawlessly, we use QML to build the front-end and it's translated in XML).

While I was designing the tests architecture, anyway, I had a doubt: how can I guarantee that every single test scenario is self-contained and totally independent from the others?

In web application testing this is not a big deal since the front-end is usually divided among multiple html pages and urls. The "given" preconditions can be easily guaranteed by requesting the right url and, usually, in the worsts scenarios, it's just needed to execute a few instructions.

With desktop applications, however, it's a totally different story. You can't just ask for the page you want to comply with the preconditions. To get to a certain state you may need to execute multiple commands. In our case many of these commands require 30+ seconds to be executed. It seems like that the best way to test every feature, with their scenarios, is to execute them one after another, in a precise order. But that's totally against the testing principles.

What I am missing? Which are the best practices in this case? Is just selenium too restrictive for our needs?

Answer 1

As an alternative to feature-based testing, you can also test your SUT based on processes/use cases. Another way is to group coherent test cases into test suites. Then, it's OK to have dependencies between test cases, but test suites should still be independent.

However, IMHO, all these things are suboptimal—for all the reasons you've already mentioned. Personally, I prefer virtualization. For instance, Vagrant is an awesome tool for testing; have a look at snapshots:

Snapshots record a point-in-time state of a guest machine. You can then quickly restore to this environment. This lets you experiment and try things and quickly restore back to a previous state.

Moreover, snapshots are often way faster (and more reliable) than complicated reset scripts. There're also many blog posts on the web regarding the combination of Vagrant and Selenium.

Answer 2

Squish for Qt supports testing of applications using QML and QWidgets including HTML content provided through QtWebKit and QWebKit. The tool was developed by former Qt (Trolltech) engineers and is used as part of the Qt release process for testing QtCreator and the example applications.

Support for is build into the IDE which means that you you can even record step implementations without starting with manual scripting right away.

To ensure that your Scenarios don't have hidden dependencies use the --random switch of squishrunner.

You are right about the time it can take to bring a desktop GUI application into the right state. One solution for that: keep the application running between and take a fast path to the required start state. This works by using attachToApplication() instead of startApplication().

(Disclaimer: I'm working with the vendor)