Automation Testing Tool Options for Desktop Applications

Question

I'm in the process of researching solutions for desk application testing for a product my company developed. There's no API, so I'm looking for other software solutions. I didn't see any recent posts on the subject, so I figured I'd start one. I need to be able to...

1. Run the software
2. Select options within the menu choices
3. Insert text into text boxes and tables
4. Mouse clicking (or some form of selecting) on submenu options, checkboxes, buttons, & other objects
5. Menu and object verification

I've done some looking around online. Here are some of the testing tools I've come across:

Sikuli
TestComplete
AutoIT
Marathon
Ranorex

- Won't Work -
--------------
PyAutoGUI, Pywinauto, Twin
- only Microsoft Windows GUI

Winium
-- Selenium Remote Control deprecated??
-- Windows applications only

I'd prefer to use Python, but I'm open to using another language or something that has its own unique programming language. Something that's open source would be preferable, but I'm open to something commercial as well. Does anyone have any suggestions on which of these would be my best option? Are there any recommendations not on this list that I should look into?

Answer 1

I have experience with Sikuli, TestComplete, and have evaluated Ranorex.

TestComplete and Ranorex are about the same price range. Of the two, I was able to get TestComplete to interact with my AUT faster than Ranorex. Both have a UI-based scripting front end that can be augmented with scripts in your language of choice. TestComplete scripts can be written in Python. Ranorex uses C# & VB.NET.

TestComplete's documentation and help is not very good. I've had more luck searching answers to TestComplete questions by Googling than by searching their own documentation. The TestComplete forums are a richer source of useful information than their formal documentation.

Sikuli is dead-easy to get started with and it's free, but it has the disadvantages of being resolution-specific and will not run on a locked machine. This means that if you create your scripts in one screen resolution and then move them to another computer with another screen resolution to execute nightly, they will fail. If you develop on one version of Windows and execute on another, the tests will fail. The inability to run on a locked machine represents a security hole unless the machine you're running on is physically inaccessible. With that said it's great for quick & dirty throwaway scripts for simple automation.

Some of the others I considered:

-- Squish by Frog Logic https://www.froglogic.com/squish/ - Works with third-party controls, does data-driven testing, uses Python for scripting, and does record/playback. On the downside it's a little expensive.

-- Eggplant by TestPlant Ltd. https://www.testplant.com/products/eggplant-functional/ - Same functionality as Sikuli, for $10k/seat/yr.

-- Winium https://github.com/2gis/Winium - Open source based on Selenium Webdriver. No GUI interface, low adoption rate.

-- Tricentis Tosca https://www.tricentis.com- Call for pricing. Web-based tool only.

-- Telerik Test Studio https://www.telerik.com/teststudio- Record and playback with scripting in C# & VB.NET. Requires Visual Studio

-- T-Plan Robot www.t-plan.com - Call for pricing. Automates from the GUI with image recognition, similar to Sikuli.

-- PyWinAuto http://pywinauto.github.io/ - Open source python library. No GUI interface, does not support some third-party controls.

-- Micro Focus SilkTest and UFT https://www.microfocus.com/products/silk-portfolio/silk-test/ - Call for pricing. Unable to glean enough info from their website to make an informed decision.

-- IBM Rational Functional Tester https://www.ibm.com/us-en/marketplace/rational-functional-tester - Call for pricing. Scripted in Java or VB.NET, with screenshot-based controls.

-- AutoIT https://www.autoitscript.com/site/autoit/ - Only works with standard Microsoft controls, does not work with third-party controls. Programmable in a proprietary BASICish language. Scripts can be compiled to run as standalone.

-- Appium Desktop https://github.com/appium/appium-desktop- Open source, drives windows via Selenium WebDriver. Scripting in Python or C#.

-- Linux Desktop Testing https://ldtp.freedesktop.org/wiki- Open source, runs on Windows/Linux.

Answer 2

Since you rule out tools that are limited to Windows GUI you seem to be in need for a multi-platform tool. Maybe even a tool that can deal with non-standard UI controls.

In case your application developers based the GUI on their own set of controls (without standard APIs) there are two possible approaches to make the automation work:

1. Direct access to properties and functions of the underlying programming framework. Like .NET.

2. Use image-based testing as well as OCR.

May I add Squish GUI Tester to your list of tools to evaluate? It is made by my company and provides above options plus Python bindings for both classic test scripts as well as Behavior-Driven Testing.