The Sikuli website says:
Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots).
Under what circumstances is Sikuli a better technology for test automation than Selenium or Watir, which depend on symbolic identifiers or paths for referencing UI elements? Please consider not only test creation but also test maintenance.
I can come up with some ideas:
You are testing for devs who don't know how to give their elements IDs or use other APIs, and won't go back and fix things in a timely fashion. This is accumulating technical debt - better to get the devs to do it right - but, e.g., on legacy apps designed without a thought toward testability, screenshot testing might be the only reasonable option for creating UI tests. In fact, this tool might solve an upcoming problem for me along those lines . . .
You are testing specifically to make sure that the UI stays reasonably similar in appearance. Normal UI testing doesn't care what the button looks like, and couldn't catch if the image used got deleted by accident or swapped with an old button image. I've seen other screenshot testing used for these purposes on major apps.
If you can get really accurate screenshots from the PM, you could make your UI tests before the IDs have been assigned to the elements and also ensure that the final UI matches PM specs, automatically.
As long as you can assign screenshots to variables and reuse them (so you only have one screenshot of the "Okay" button in the entire test suite), maintenance wouldn't be too bad IMO. It might even be easier if IDs are still changing and the UI is in extreme flux. I worked on a project once where devs kept removing IDs by accident, and I had to wait literally weeks before I got a reliable identifier quite regularly. But UI elements will always have an image, by definition, meaning a 1-minute test fix rather than a 3-week wait where I can't run that test automatically and need to run it manually, or a 15-minute to 1-hour fix to code a work-around function and test it.
The main difficulty I can think of would arise when there could be two or more similar pieces of UI (e.g., two "Okay" buttons, one on an error and one on a standard configuration screen), but even IDs can have similar issues (do you want "dynamicMenuOption11" or "dynamicMenuOption12"?), and you could probably fix this most of the time just by taking a bigger screenshot (e.g., get the edge of the "Apply" button or error dialogue to identify the right button). That's an easier fix than coding the ID in, checking other values, and so on, IMO.
Edit: I do think Tarun brings up a good point about how well Sikuli tolerates changes. The video claims some sort of "best-fit" matching, but how tolerant is that? And, if it is too tolerant, could it produce false actions and false passes?
I have actually used Sikuli and believe it is the future for QA automation. The fuzzy logic on image compare is pure genius. You can crank it up to be an exact match or turn it down to lesser. I have ran the same test on multiple browsers/OSes and resolutions without too many issues.
It is way more robust than the old x,y testing of yesteryear. I created an image.sikuli script that hosts all my images and I just reuse them through out.
The app I am tasked with testing does not play well with QTP or TestComplete, I am unable to see elements. This seems to be more and more common with applications that are using embedded components. Sikuli bridges that gap, and I find it easier to use than the others (although the IDE is not very good). It does not have the record and playback that some of the other tools promote, but again doing that never produces a repeatable robust automation.
Selenium also does not do ANY image compare. Although the test may pass, it may look like crap on IE 9. Sikuli can at least do a high level compare on elements, and buttons to make sure they are rendering correctly.
In my opinion, none!
When you are using images to locate elements then you are in trouble when someone comes along and decides to change how the application looks or the position of an element on the screen. You will build a fragile test suite and will be forever updating the images that you have used.
One concern would be that IE and firefox don't always render css in the same way and subtle differences would cause your test to fail, so you would need to be checking for different images in your test. This is really an overhead that you could save yourself from doing.
Selenium and Watir on the other hand doesn't care what the element looks like, instead you can find it using many different techniques (id, name, class...) that allow you to build a reliable test suite.
I am not sure it's as easy as saying Sikuli is a better technology for test automation than Selenium or Watir - it's just an alternative.
On my current project, the front end test team were struggling to test an application using flexmonkium. They had the skills and experience to write tests in flexmonkium, but our application needed to be recompiled with the flexmonkium libraries, otherwise the tests simply couldn't hook into the application. It was not going to be possible to make this happen for quite a while, so the team needed another approach. We choose Sikuli, since it didn't need any hooks into the application under test - it simply works off what is on screen.
Sikuli has drawbacks (slow, unpredictable, cross browser issues etc.), but in some scenarios, it may be a good choice. You need to make up your own mind.
Sikuli is useful whenever you're automating something that's not web based.
I really like Selenium and Watir, but they're restricted to web technologies, and even though these are probably the most common nowadays, there are still plenty of companies who use desktop apps. For these, I find Sikuli and its fuzzy matches to be quite useful.
Bear in mind that the IDE is not great -- and that you'll need to use it to generate the screenshots and adjust matching precision and position.
I have used it for 1 day. I think it is easier to use and faster to learn. Also you have the benefits of image compare. (Selenium does not have this). This said, I think You could use it for Flash automation. (I never managed to do this in a good way with selenium).
Furthermore you don't have to learn autoIt or another scripting for desktop automation.
So it has some advantages. But in my opinion you are better of learning selenium and autoit, and maybe use sikuli as a plugin.
When starting from scratch with little programming experience and you need to have fast results. (or for demo purposes). You could use it, especially when you need desktop automation / scripting as well.
When you only need webscripting you could use imacros as well, since this can be picked up quickly.
I've used Sikuli several times and my big win was for testing memory leaks on our mobile app (using the simulator) that i could not have found manually. Also running with Sikuli allows us to bypass budget restrictions :) Additionally we have customized apps, so i like that i can use Sikuli to make sure the right colour is applied in the right place - trust me i forget which client gets which label or which colour where, so i love it for that.
It's pretty easy to learn (IMHO) - i have no coding background and i am able to use it. The user forums rock - they will help and provide answers in a very short timeframe.
My company has the standard-issue (at least these days) automation buzzword stack of Maven, Selenium, Jenkins, blah-blah and so on. I use Sikuli. What it takes them a week to automate I automate in a couple hours.
Sikuli can automate things Selenium simply cannot, such as Flash or iPhone sims. Selenium can check if a video file is there, but Sikuli can play it and 'watch' it. I see, especially with web pages, Selenium automators constantly battling page loads with all kinds of wait-this-and-wait-that, leaving the script sitting there for long periods doing nothing and at other times failing out while a page is still loading. Sikuli has event handlers (onChange(), onVanish(), etc.) which if used with any cleverness will handle the tests dynamic to the connection simulated, literally waiting like a user for things to happen and then acting on them.
Same with finding correct objects to test. A couple parameters passed into a loop and Sikuli will go 'find' the object to test just like a user, and likewise know when to give up. I see Selenium folks resorting to passing database hacks to find the same object - hardly an automated user-acceptance test!
Granted, to use the tool successfully Python skills are needed, but everyone knows Java or Ruby given they are the current buzzword-shiny things managers are chasing. And Python is nothing like those languages. And of course there is the risk of images changing - which is nowhere near as bad as you think (when iOS7 actually launches, I've got about a half-day of work updating images). But when used appropriately, my experience leads me to concluding Sikuli smokes Selenium.
Here some points about sikuli I have discovered:
Sikuli can not select by 100% text match. So it is extremely difficult to use it in financial applications cause there are not too many different GUI elements, but lots of similar buttons, lists, tables... So how could you select specific string (by its contents) in table with 1000+ rows (only 25 are in a visible area, the question about lists & tables) with tool that works by text recognition. Even if you will find the way to scroll to the item you need here comes the problem to distinguish row with "Economic Community of West African States" from "Economic Community of East African States". With maximum recognition strength 0,99 they are still the same.
Sikuli is unpredictable. Some GUI elements can change their skins, colors, images (here is an example of transparency in windows GUI, so the autotest was broken just because of changed background):
Sikuli is fast to learn. Need just half of a day to create the first test (navigation through several windows, selecting values in drop down lists, generating report)
Sikuli tests look like comix. In native Sikuli IDE test looks like comix, someone can like that but I prefer text instructions :)
I have used Sikuli and found it to be a great help in testing WPF.NET GUI applications (which do not comprise standard window instances with window handles like other Window apps do) and are otherwise very hard to test. I think Watir and Selenium are primarily for testing browser based applications, whereas Sikuli can also be used to test non-browser GUI applications.
I think putting all your money on image recognition only tests might get you into trouble as soon as the look/layout of a site changes and you didn't or couldn't make your screenshot collection smart enough to still work - in that case all your test cases would need to be repaired. I'd try a combination of both - Sikuli as well as Selenium are available as Java libraries - combine it with JUnit and you're set. Write as much as possible in Selenium but use Sikuli for visual confirmation or for hard to implement stuff. (don't know about web design much, but in Java software with some test software only tools like Sikuli managed to do drag & drop properly)
I am using Selenium in my project for functional testing and because Selenium doesn't support GUI testing so for that I am using Sikuli to capture GUI defects so for me both are complimenting each other if used properly. Both sikuli and selenium has its pros n cons. It depends on which kind of web application you are testing and objective of testing. Sikuli has its limitation to capture functional defetcs on multi browser testing as all browsers renders differently. And Selenium don't has any feature to capture GUI defects. Only thing we capture a screenshot and check manually GUI defects. So instead of chechking manually I am using sikuli in our project. As requirement is to test multi-lingual, multi browser testing so all functional things I am covering with Selenium using Java language in Eclipse IDE. And capturing snapshots, then using Sikuli to find GUI defects. And its easy to integrate Sikuli with java because Sikuli itself is a JAR file which we can easily set in our eclipse buid path.
I just watched their video and answering this on basis of certain assumption I made, Sikuli is better when -
Tests is only on one browser, one operating system, one resolution, one (every thing else you could think of) OR if you are ready to develop different Sikuli scripts for each configuration.
Sikuli system some how reminds me of old time automation tool when UI operations would be tied to screen coordinates. And then we moved to better element locators for good reasons.
Though if you see the intro video of Sikuli for mac, I feel Sikuli is better off for these task then browser basesd UI automation in Agile environment.
Sikuli is useful when there is no easy access to a GUI's internal or source code. Selenium or Watir identifies web elements by ID, Name, XPath etc. on the contrary Sikuli finds the elements by image/screenshot which is easier.
I prefer Sikuli for quick tests of a specific functionality/defect, and then Selenium for regression/catch-all E2E tests.
Selenium:
The functionality of questionnaire. Checks for UI text based on ids, and then runs through all permutations of the questionnaire to check that all behavior and page redirects are as expected.
Run through --suite=all at the end of each story to make sure nothing was broken elsewhere in the app. It's quick and runs in the background, and can test in multiple browsers at the same time.
Coding is done in Protractor, which compliments our AngularJS app.
Seems more stable and refined to me, with much better documentation and resources online if something were to act up.
Minor point, but it's much easier to show "real" work is being done in Javascript as opposed to the Sikuli style of code. Devs and higher ups see it as more real engineering being done.
Sikuli:
Quick metrics for running through a process as a real user would. I can very quickly put together a script that will use Python to keep track of how long the script is running at different points, then have my mouse taken over and the app gone through like a real person would. Not milliseconds like Selenium.
Sikuli isn't just for screenshots! On a previous project when I did the above for my manager, I used regions instead of screenshots. It helped since our lists we needed to click on were pulled from an api, and the source changed frequently. Made that script useful regardless of the tile it was choosing (I just had it pick the first in the list)
If there's a front end ready but identifiers aren't in place yet/at all, Sikuli doesn't care and just goes to work. You don't need access to the code at all, which works well if you're distant from the developers.
We once had a defect that was easy enough to reproduce, but the only issue it caused was that our grids overlapped and just looked wrong. Sikuli made it easier to look for a fuzzy image of the overlapping and check if it was still doing that each build.
Minor, but Python is a very easy language to learn, and is supported natively by Sikuli. Your tests can be a hybrid of the click(picture1.png) and Python to have a wide range of functionality.
I personally think Selenium is better for test maintenance, but Sikuli makes test creation a breeze. Also, Sikuli better mimics how a user would go through the app since it takes over the mouse rather than running in the background. Selenium strikes me as much better for larger work and a professional setting. Time and a place for both, I suppose.
I use Selenium (through CodeCeption) and it works well when I can target some IDs or CSS rules.
But we have Canvas & Flash games which expose nothing for testing. The only interface is the GUI !
These games have "events" to which my test has to respond correctly to resume the main "game" (Yep, the DEV didn't thought of test-ability!)
For these, I use Sikuli. It uses Python for scripts and it is possible to disable the display of image AND get him ask a name for each taken screenshot.
For handling "events" I use simple "try catch" constructs so my test can run through a game happily.
The longest part is reverse engineering since, of course, games aren't documented.
I have the same speed scripting in CodeCeption than Sikuli, if it takes more time in Sikuli, this is only due because I can't force these developers to provide interfaces for testing etc.
For the website, I ask IDs and classes on key elements (form elements, any interractable, outputs, ... )
If I fire Sikuli, someone did something wrong :-D
If you didn't pay attention to circumstances above :
A side feature is the automatic generation of the HTML file describing the procedure. Project managers understand them ! (Ther is a lot of images...)
