|
The Sikuli website says:
Under what circumstances is Sikuli a better technology for test automation than Selenium or Watir, which depend on symbolic identifiers or paths for referencing UI elements? Please consider not only test creation but also test maintenance. |
||||
|
|
I can come up with some ideas:
As long as you can assign screenshots to variables and reuse them (so you only have one screenshot of the "Okay" button in the entire test suite), maintenance wouldn't be too bad IMO. It might even be easier if IDs are still changing and the UI is in extreme flux. I worked on a project once where devs kept removing IDs by accident, and I had to wait literally weeks before I got a reliable identifier quite regularly. But UI elements will always have an image, by definition, meaning a 1-minute test fix rather than a 3-week wait where I can't run that test automatically and need to run it manually, or a 15-minute to 1-hour fix to code a work-around function and test it. The main difficulty I can think of would arise when there could be two or more similar pieces of UI (e.g., two "Okay" buttons, one on an error and one on a standard configuration screen), but even IDs can have similar issues (do you want "dynamicMenuOption11" or "dynamicMenuOption12"?), and you could probably fix this most of the time just by taking a bigger screenshot (e.g., get the edge of the "Apply" button or error dialogue to identify the right button). That's an easier fix than coding the ID in, checking other values, and so on, IMO. Edit: I do think Tarun brings up a good point about how well Sikuli tolerates changes. The video claims some sort of "best-fit" matching, but how tolerant is that? And, if it is too tolerant, could it produce false actions and false passes? |
|||||
|
|
I have actually used Sikuli and believe it is the future for QA automation. The fuzzy logic on image compare is pure genius. You can crank it up to be an exact match or turn it down to lesser. I have ran the same test on multiple browsers/OSes and resolutions without too many issues. It is way more robust than the old x,y testing of yesteryear. I created an image.sikuli script that hosts all my images and I just reuse them through out. The app I am tasked with testing does not play well with QTP or TestComplete, I am unable to see elements. This seems to be more and more common with applications that are using embedded components. Sikuli bridges that gap, and I find it easier to use than the others (although the IDE is not very good). It does not have the record and playback that some of the other tools promote, but again doing that never produces a repeatable robust automation. Selenium also does not do ANY image compare. Although the test may pass, it may look like crap on IE 9. Sikuli can at least do a high level compare on elements, and buttons to make sure they are rendering correctly. |
|||
|
|
|
In my opinion, none! When you are using images to locate elements then you are in trouble when someone comes along and decides to change how the application looks or the position of an element on the screen. You will build a fragile test suite and will be forever updating the images that you have used. One concern would be that IE and firefox don't always render css in the same way and subtle differences would cause your test to fail, so you would need to be checking for different images in your test. This is really an overhead that you could save yourself from doing. Selenium and Watir on the other hand doesn't care what the element looks like, instead you can find it using many different techniques (id, name, class...) that allow you to build a reliable test suite. |
|||||||||||||||||
|
|
I have used it for 1 day. I think it is easier to use and faster to learn. Also you have the benefits of image compare. (Selenium does not have this). This said, I think You could use it for Flash automation. (I never managed to do this in a good way with selenium). Furthermore you don't have to learn autoIt or another scripting for desktop automation. So it has some advantages. But in my opinion you are better of learning selenium and autoit, and maybe use sikuli as a plugin. When starting from scratch with little programming experience and you need to have fast results. (or for demo purposes). You could use it, especially when you need desktop automation / scripting as well. When you only need webscripting you could use imacros as well, since this can be picked up quickly. |
|||
|
|
I am not sure it's as easy as saying Sikuli is a better technology for test automation than Selenium or Watir - it's just an alternative. On my current project, the front end test team were struggling to test an application using flexmonkium. They had the skills and experience to write tests in flexmonkium, but our application needed to be recompiled with the flexmonkium libraries, otherwise the tests simply couldn't hook into the application. It was not going to be possible to make this happen for quite a while, so the team needed another approach. We choose Sikuli, since it didn't need any hooks into the application under test - it simply works off what is on screen. Sikuli has drawbacks (slow, unpredictable, cross browser issues etc.), but in some scenarios, it may be a good choice. You need to make up your own mind. |
|||
|
|
|
Sikuli is useful whenever you're automating something that's not web based. I really like Selenium and Watir, but they're restricted to web technologies, and even though these are probably the most common nowadays, there are still plenty of companies who use desktop apps. For these, I find Sikuli and its fuzzy matches to be quite useful. Bear in mind that the IDE is not great -- and that you'll need to use it to generate the screenshots and adjust matching precision and position. |
|||||||
|
|
I've used Sikuli several times and my big win was for testing memory leaks on our mobile app (using the simulator) that i could not have found manually. Also running with Sikuli allows us to bypass budget restrictions :) Additionally we have customized apps, so i like that i can use Sikuli to make sure the right colour is applied in the right place - trust me i forget which client gets which label or which colour where, so i love it for that. It's pretty easy to learn (IMHO) - i have no coding background and i am able to use it. The user forums rock - they will help and provide answers in a very short timeframe. |
|||||
|
|
I think putting all your money on image recognition only tests might get you into trouble as soon as the look/layout of a site changes and you didn't or couldn't make your screenshot collection smart enough to still work - in that case all your test cases would need to be repaired. I'd try a combination of both - Sikuli as well as Selenium are available as Java libraries - combine it with JUnit and you're set. Write as much as possible in Selenium but use Sikuli for visual confirmation or for hard to implement stuff. (don't know about web design much, but in Java software with some test software only tools like Sikuli managed to do drag & drop properly) |
|||
|
|
I have used Sikuli and found it to be a great help in testing WPF.NET GUI applications (which do not comprise standard window instances with window handles like other Window apps do) and are otherwise very hard to test. I think Watir and Selenium are primarily for testing browser based applications, whereas Sikuli can also be used to test non-browser GUI applications. |
|||
|
|
|
I am using Selenium in my project for functional testing and because Selenium doesn't support GUI testing so for that I am using Sikuli to capture GUI defects so for me both are complimenting each other if used properly. Both sikuli and selenium has its pros n cons. It depends on which kind of web application you are testing and objective of testing. Sikuli has its limitation to capture functional defetcs on multi browser testing as all browsers renders differently. And Selenium don't has any feature to capture GUI defects. Only thing we capture a screenshot and check manually GUI defects. So instead of chechking manually I am using sikuli in our project. As requirement is to test multi-lingual, multi browser testing so all functional things I am covering with Selenium using Java language in Eclipse IDE. And capturing snapshots, then using Sikuli to find GUI defects. And its easy to integrate Sikuli with java because Sikuli itself is a JAR file which we can easily set in our eclipse buid path. |
|||
|
|
|
I just watched their video and answering this on basis of certain assumption I made, Sikuli is better when - Tests is only on one browser, one operating system, one resolution, one (every thing else you could think of) OR if you are ready to develop different Sikuli scripts for each configuration. Sikuli system some how reminds me of old time automation tool when UI operations would be tied to screen coordinates. And then we moved to better element locators for good reasons. Though if you see the intro video of Sikuli for mac, I feel Sikuli is better off for these task then browser basesd UI automation in Agile environment. |
|||||
|
