I have experience with Sikuli, TestComplete, and have evaluated Ranorex.
TestComplete and Ranorex are about the same price range. Of the two, I was able to get TestComplete to interact with my AUT faster than Ranorex. Both have a UI-based scripting front end that can be augmented with scripts in your language of choice. TestComplete scripts can be written in Python. Ranorex uses C# & VB.NET.
TestComplete's documentation and help is not very good. I've had more luck searching answers to TestComplete questions by Googling than by searching their own documentation. The TestComplete forums are a richer source of useful information than their formal documentation.
Sikuli is dead-easy to get started with and it's free, but it has the disadvantages of being resolution-specific and will not run on a locked machine. This means that if you create your scripts in one screen resolution and then move them to another computer with another screen resolution to execute nightly, they will fail. If you develop on one version of Windows and execute on another, the tests will fail. The inability to run on a locked machine represents a security hole unless the machine you're running on is physically inaccessible. With that said it's great for quick & dirty throwaway scripts for simple automation.
Some of the others I considered:
-- Squish by Frog Logic https://www.froglogic.com/squish/ - Works with third-party controls, does data-driven testing, uses Python for scripting, and does record/playback. On the downside it's a little expensive.
-- Eggplant by TestPlant Ltd. https://www.testplant.com/products/eggplant-functional/ - Same functionality as Sikuli, for $10k/seat/yr.
-- Winium https://github.com/2gis/Winium - Open source based on Selenium Webdriver. No GUI interface, low adoption rate.
-- Tricentis Tosca https://www.tricentis.com- Call for pricing. Web-based tool only.
-- Telerik Test Studio https://www.telerik.com/teststudio- Record and playback with scripting in C# & VB.NET. Requires Visual Studio
-- T-Plan Robot www.t-plan.com - Call for pricing. Automates from the GUI with image recognition, similar to Sikuli.
-- PyWinAuto http://pywinauto.github.io/ - Open source python library. No GUI interface, does not support some third-party controls.
-- Micro Focus SilkTest and UFT https://www.microfocus.com/products/silk-portfolio/silk-test/ - Call for pricing. Unable to glean enough info from their website to make an informed decision.
-- IBM Rational Functional Tester https://www.ibm.com/us-en/marketplace/rational-functional-tester - Call for pricing. Scripted in Java or VB.NET, with screenshot-based controls.
-- AutoIT https://www.autoitscript.com/site/autoit/ - Only works with standard Microsoft controls, does not work with third-party controls. Programmable in a proprietary BASICish language. Scripts can be compiled to run as standalone.
-- Appium Desktop https://github.com/appium/appium-desktop- Open source, drives windows via Selenium WebDriver. Scripting in Python or C#.
-- Linux Desktop Testing https://ldtp.freedesktop.org/wiki- Open source, runs on Windows/Linux.