Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

The Sikuli website says:

Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots).

Under what circumstances is Sikuli a better technology for test automation than Selenium or Watir, which depend on symbolic identifiers or paths for referencing UI elements? Please consider not only test creation but also test maintenance.

share|improve this question
    
While I don't use Sikuli for test automation, I have found that a Sikuli script that reproduces a bug can be run in less time than it takes for a Dev to follow a set of steps to reproduce the same bug. Of course, YMMV. –  user867 May 11 '12 at 2:13

13 Answers 13

up vote 8 down vote accepted

I can come up with some ideas:

  • You are testing for devs who don't know how to give their elements IDs or use other APIs, and won't go back and fix things in a timely fashion. This is accumulating technical debt - better to get the devs to do it right - but, e.g., on legacy apps designed without a thought toward testability, screenshot testing might be the only reasonable option for creating UI tests. In fact, this tool might solve an upcoming problem for me along those lines . . .

  • You are testing specifically to make sure that the UI stays reasonably similar in appearance. Normal UI testing doesn't care what the button looks like, and couldn't catch if the image used got deleted by accident or swapped with an old button image. I've seen other screenshot testing used for these purposes on major apps.

  • If you can get really accurate screenshots from the PM, you could make your UI tests before the IDs have been assigned to the elements and also ensure that the final UI matches PM specs, automatically.

As long as you can assign screenshots to variables and reuse them (so you only have one screenshot of the "Okay" button in the entire test suite), maintenance wouldn't be too bad IMO. It might even be easier if IDs are still changing and the UI is in extreme flux. I worked on a project once where devs kept removing IDs by accident, and I had to wait literally weeks before I got a reliable identifier quite regularly. But UI elements will always have an image, by definition, meaning a 1-minute test fix rather than a 3-week wait where I can't run that test automatically and need to run it manually, or a 15-minute to 1-hour fix to code a work-around function and test it.

The main difficulty I can think of would arise when there could be two or more similar pieces of UI (e.g., two "Okay" buttons, one on an error and one on a standard configuration screen), but even IDs can have similar issues (do you want "dynamicMenuOption11" or "dynamicMenuOption12"?), and you could probably fix this most of the time just by taking a bigger screenshot (e.g., get the edge of the "Apply" button or error dialogue to identify the right button). That's an easier fix than coding the ID in, checking other values, and so on, IMO.

Edit: I do think Tarun brings up a good point about how well Sikuli tolerates changes. The video claims some sort of "best-fit" matching, but how tolerant is that? And, if it is too tolerant, could it produce false actions and false passes?

share|improve this answer
1  
The tolerance is adjustable, but it's default value is '0.7.' I've found that is a little too tolerant for the application I've been testing, but one of the longstanding bugs in the application I've been testing is that it's got very poor visual contrast. Make of that what you will. –  user867 May 11 '12 at 2:05

I agree, Sikuli alone isn't do the trick. You have to combine many testing techniques. You have to use every tool available to test your software, mobile or desktop, and your website as good and hard as you can.

I recently learned if you develop software for embedded devices they use all available tricks to test their deviced, so they are fool proof. So why not combining sikuli with selenium or integrate sikuli into junit maven tests. I don't see a problem here.

I for myself have decided to give Sikuli a go and I wrote a litte BDD Testframework "JePySi" to test my websites. I created a couple of standard VMs with jenkins slaves and the run the tests triggered by a CI/CD job construct on a Jenkins master. And so far I'm very pleased with it.

Of course you have to overcome some obstacles. The images eg. Retina screenshots do not work at all with sikuli. You have to take screenshots with the Sikuli IDE, so they work. And Yes sometimes scripts fail. But most of the time, because there are OS differences, app differences or rendering differences. But that's the challenge I got mostly tackled with my framework.

E.g. allow different images on different OS enviroments for the same image. E.g. the green SSL header from bitbucket.org is rendered differently on mac and windows. So I give them a name like "bitbucket.org header" and in the background I've two images one with the mac rendering and one with the windows or linux image. And I wrote a complete Testsuite for jepysi.arjs.net within one day.

Yes sikuli is great for apps and websites and Yes selenium is also a very good tool for website testing and not so much for apps. In my opinion you should use what you think get the job done. I prever sikuli for now.

Btw. I open sourced "JePySi" on bitbucket.org, since I believe in open source. So if you interessted give it a spin.

Cheers, Alex

share|improve this answer
1  
Thank you user6917, but that was not answer to the question. –  user246 Feb 5 at 11:56

My company has the standard-issue (at least these days) automation buzzword stack of Maven, Selenium, Jenkins, blah-blah and so on. I use Sikuli. What it takes them a week to automate I automate in a couple hours.

Sikuli can automate things Selenium simply cannot, such as Flash or iPhone sims. Selenium can check if a video file is there, but Sikuli can play it and 'watch' it. I see, especially with web pages, Selenium automators constantly battling page loads with all kinds of wait-this-and-wait-that, leaving the script sitting there for long periods doing nothing and at other times failing out while a page is still loading. Sikuli has event handlers (onChange(), onVanish(), etc.) which if used with any cleverness will handle the tests dynamic to the connection simulated, literally waiting like a user for things to happen and then acting on them.

Same with finding correct objects to test. A couple parameters passed into a loop and Sikuli will go 'find' the object to test just like a user, and likewise know when to give up. I see Selenium folks resorting to passing database hacks to find the same object - hardly an automated user-acceptance test!

Granted, to use the tool successfully Python skills are needed, but everyone knows Java or Ruby given they are the current buzzword-shiny things managers are chasing. And Python is nothing like those languages. And of course there is the risk of images changing - which is nowhere near as bad as you think (when iOS7 actually launches, I've got about a half-day of work updating images). But when used appropriately, my experience leads me to concluding Sikuli smokes Selenium.

share|improve this answer

I am using Selenium in my project for functional testing and because Selenium doesn't support GUI testing so for that I am using Sikuli to capture GUI defects so for me both are complimenting each other if used properly. Both sikuli and selenium has its pros n cons. It depends on which kind of web application you are testing and objective of testing. Sikuli has its limitation to capture functional defetcs on multi browser testing as all browsers renders differently. And Selenium don't has any feature to capture GUI defects. Only thing we capture a screenshot and check manually GUI defects. So instead of chechking manually I am using sikuli in our project. As requirement is to test multi-lingual, multi browser testing so all functional things I am covering with Selenium using Java language in Eclipse IDE. And capturing snapshots, then using Sikuli to find GUI defects. And its easy to integrate Sikuli with java because Sikuli itself is a JAR file which we can easily set in our eclipse buid path.

share|improve this answer

Sikuli is useful whenever you're automating something that's not web based.

I really like Selenium and Watir, but they're restricted to web technologies, and even though these are probably the most common nowadays, there are still plenty of companies who use desktop apps. For these, I find Sikuli and its fuzzy matches to be quite useful.

Bear in mind that the IDE is not great -- and that you'll need to use it to generate the screenshots and adjust matching precision and position.

share|improve this answer
    
+1, I was just going to comment on this. Nobody else seems to mention that Sikuli and like tools work on desktop and native mobile apps (mobile via emulator) whereas Selenium and Watir are web only. –  David Jul 13 '12 at 22:38
    
One other comment, you're not restricted to using the Sikuli IDE to generate screenshots nor setting precision and positioning. The IDE like Selenium IDE is for throwaway code/tests, and for beginners. There is a robust Java and Python API you can use to build your tests. And you can also manually create your reference image screenshots, just save as PNG format, or call code to dynamically convert from source format to PNG before passing to Sikuli. –  David Jul 13 '12 at 22:40
    
Yeah, I believe my comment was a bit too deterministic about needing the IDE for the screenshots and adjustments. I still find the IDE useful for those adjustments, though. –  cbl Jul 14 '12 at 16:28

I have used Sikuli and found it to be a great help in testing WPF.NET GUI applications (which do not comprise standard window instances with window handles like other Window apps do) and are otherwise very hard to test. I think Watir and Selenium are primarily for testing browser based applications, whereas Sikuli can also be used to test non-browser GUI applications.

share|improve this answer
    
again +1 about the desktop testing support. –  David Jul 13 '12 at 22:41

I think putting all your money on image recognition only tests might get you into trouble as soon as the look/layout of a site changes and you didn't or couldn't make your screenshot collection smart enough to still work - in that case all your test cases would need to be repaired. I'd try a combination of both - Sikuli as well as Selenium are available as Java libraries - combine it with JUnit and you're set. Write as much as possible in Selenium but use Sikuli for visual confirmation or for hard to implement stuff. (don't know about web design much, but in Java software with some test software only tools like Sikuli managed to do drag & drop properly)

share|improve this answer
    
This. Test automation is all about efficiency, and using each tool for what it's most suited for can be much more efficient than trying to find a one-size-fits-all solution. –  user867 Oct 24 '12 at 4:55

I've used Sikuli several times and my big win was for testing memory leaks on our mobile app (using the simulator) that i could not have found manually. Also running with Sikuli allows us to bypass budget restrictions :) Additionally we have customized apps, so i like that i can use Sikuli to make sure the right colour is applied in the right place - trust me i forget which client gets which label or which colour where, so i love it for that.

It's pretty easy to learn (IMHO) - i have no coding background and i am able to use it. The user forums rock - they will help and provide answers in a very short timeframe.

share|improve this answer
    
Can you explain how you used Sikuli to test memory leaks? –  user246 Aug 30 '11 at 19:32
    
in the iOS simulator there is a "simulate low memory warning" so i had a simple workflow that i had done as proof of concept for running Sikuli on iOS simulator so i added a function and then called it throughout the script... then looped it 100 times :) worked like a charm –  qualitycurls Aug 30 '11 at 19:57

I am not sure it's as easy as saying Sikuli is a better technology for test automation than Selenium or Watir - it's just an alternative.

On my current project, the front end test team were struggling to test an application using flexmonkium. They had the skills and experience to write tests in flexmonkium, but our application needed to be recompiled with the flexmonkium libraries, otherwise the tests simply couldn't hook into the application. It was not going to be possible to make this happen for quite a while, so the team needed another approach. We choose Sikuli, since it didn't need any hooks into the application under test - it simply works off what is on screen.

Sikuli has drawbacks (slow, unpredictable, cross browser issues etc.), but in some scenarios, it may be a good choice. You need to make up your own mind.

share|improve this answer

I have used it for 1 day. I think it is easier to use and faster to learn. Also you have the benefits of image compare. (Selenium does not have this). This said, I think You could use it for Flash automation. (I never managed to do this in a good way with selenium).

Furthermore you don't have to learn autoIt or another scripting for desktop automation.

So it has some advantages. But in my opinion you are better of learning selenium and autoit, and maybe use sikuli as a plugin.

When starting from scratch with little programming experience and you need to have fast results. (or for demo purposes). You could use it, especially when you need desktop automation / scripting as well.

When you only need webscripting you could use imacros as well, since this can be picked up quickly.

share|improve this answer
    
I don't see how it's better learning AutoIt. Sikuli and AutoIt do about the same things GUI automation of web and desktop apps. Only difference is Sikuli is image based and AutoIt is UI component identifier based. But I have seen cases where AutoIt can't identify a desktop (not browser) UI element, so you'd have to resort to Sikuli, or commercial test tool, or make do with keyboard shortcuts or mouse movements to compensate for not being able to control some UI element. –  David Jul 13 '12 at 22:44

I have actually used Sikuli and believe it is the future for QA automation. The fuzzy logic on image compare is pure genius. You can crank it up to be an exact match or turn it down to lesser. I have ran the same test on multiple browsers/OSes and resolutions without too many issues.

It is way more robust than the old x,y testing of yesteryear. I created an image.sikuli script that hosts all my images and I just reuse them through out.

The app I am tasked with testing does not play well with QTP or TestComplete, I am unable to see elements. This seems to be more and more common with applications that are using embedded components. Sikuli bridges that gap, and I find it easier to use than the others (although the IDE is not very good). It does not have the record and playback that some of the other tools promote, but again doing that never produces a repeatable robust automation.

Selenium also does not do ANY image compare. Although the test may pass, it may look like crap on IE 9. Sikuli can at least do a high level compare on elements, and buttons to make sure they are rendering correctly.

share|improve this answer

In my opinion, none!

When you are using images to locate elements then you are in trouble when someone comes along and decides to change how the application looks or the position of an element on the screen. You will build a fragile test suite and will be forever updating the images that you have used.

One concern would be that IE and firefox don't always render css in the same way and subtle differences would cause your test to fail, so you would need to be checking for different images in your test. This is really an overhead that you could save yourself from doing.

Selenium and Watir on the other hand doesn't care what the element looks like, instead you can find it using many different techniques (id, name, class...) that allow you to build a reliable test suite.

share|improve this answer
1  
That is my impression too, but I thought I might be missing something. People use visual cues to understand how user interfaces are organized, e.g. all the text fields and buttons within a box are probably part of the same form, and the highlighted ones are probably mandatory fields. If Sikuli could work with visual cues, I think it could be as powerful as Selenium/Watir. Perhaps that is where the project is headed. –  user246 Jun 7 '11 at 14:43
    
Have to agree that i see little use for this. Screenshot comparisons are notoriously difficult for computers, but very easy for us humans. –  Ardesco Jun 30 '11 at 11:20
3  
I think some people overlook how to really use Sikuli. You don't take a screenshot of whole (web)page & work with it. That's not the point. There are screenshot tools you can use then manually compare or run some tool to compare screenshot to reference image. The way you use Sikuli is to define "atomic" UI elements (i.e. button on page, a link, some text) typically w/o dependency on layout of other elements and save cropped screenshots of them as reference images. Then compare them against what you see on screen at runtime. So you build a repository of graphic UI elements vs web UI locators. –  David Jul 13 '12 at 22:49
3  
had to break up comment due to length. So you do Sikuli tests like can you click this button on screen (doe it exist anywhere on screen)? And you can make it relative to some other elements on screen as in button must be next to these items, etc. How atomic you make it depends on your needs. Fully atomic means it can be anywhere and not relative to some other elements. Using these techniques properly, changes in page layout won't necessarily break a test. You just have to update UI elements when changed (e.g. button look) and page layout only if using relativity of elements to each other. –  David Jul 13 '12 at 22:52
    
Check out my Sikuli wrapper project that presents a bit of how to appropriately use Sikuli as well as a smaller core subset API of what you would do with Sikuli: code.google.com/p/simplesikuli/wiki/UsageDetails –  David Jul 13 '12 at 22:55

I just watched their video and answering this on basis of certain assumption I made, Sikuli is better when -

Tests is only on one browser, one operating system, one resolution, one (every thing else you could think of) OR if you are ready to develop different Sikuli scripts for each configuration.

Sikuli system some how reminds me of old time automation tool when UI operations would be tied to screen coordinates. And then we moved to better element locators for good reasons.

Though if you see the intro video of Sikuli for mac, I feel Sikuli is better off for these task then browser basesd UI automation in Agile environment.

share|improve this answer
    
It's not quite as bad as you make it sound. It depends on the application you want to test, of course, but most of the time Sikuli is only trying to match small sections of the screen, so resolution is rarely a problem, and the different appearences of browsers and operating systems can be compensated for with sufficiently robust test scripts. –  user867 May 11 '12 at 2:10
    
Agree wit user867. Screen resolution only comes into play when you want to specifically test against something like a browser window and page rendering at 1024x768. If you're matching/testing against a button or other graphic element, it's size will always be fixed regardless of the screen size, unless someone changed the zoom level of the desktop, etc. And doesn't matter about position of element on screen, Sikuli knows how to find it anywhere on the screen. –  David Jul 13 '12 at 22:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.