0

I have recently joined a company and we are looking to standardize our testing frameworks and tools across multiple (many legacy) products.

Because of this, I'm seriously considering the Selenium Test framework for automated regression of web based systems.

What are its unique advantages? Specifically, what features does Selenium posses that are unique and not present in competing frameworks?

4
  • 1
    Cucumber is not a GUI automation tool, its more a test describer and runner like xUnit. Oct 29, 2015 at 15:31
  • I have considerable experience using Selenium in various languages (Java + C# mainly). The decision to use selenium has traditionally been dictated by the fact it was the incumbent framework used for testing. Many of the limitations I have come across over the years, but finding a list of unique advantages tends to prove difficult.
    – ECiurleo
    Oct 29, 2015 at 16:45
  • Edited to make the question more focused
    – ECiurleo
    Dec 14, 2015 at 13:34
  • @BharatMane can you please stop changing format of posts? Quote sign > is to quote someone. If you want to emphasize statement you can use italics * * or bold font ** **.
    – dzieciou
    Dec 26, 2016 at 14:45

5 Answers 5

7

Although, this question is too broad to answer but in short I can say 'It depends upon number of factor which tool you want to use'.

  1. Project/Organization budget
  2. Scope of Automation
  3. Skillset available with team
  4. Complexity/functionalities of automation
  5. Reporting
  6. Test case management
  7. Tool support, etc.

Don't just compare the tools theoretically, use them for different types of applications which you want to support with that tool and then decide based on your findings which tool you want to go with. Every tool has some limitations, some advantages, some dis-advantages and some assumptions which has to be considered while selecting a tool, as this is a big decision especially when it is a licensed tool and you have to use it for number of applications and have to show ROI.

Now, coming to your question in detail. I have tried (compared) Selenium, Telerik's Test Studio and Test Complete for one of our web-application long ago (so after a detailed analysis we selected Test Studio, the only reason was that application was rich in Telerik's control, hence we selected their own testing tool and that time we supported majorly IE browser). But have found Selenium to be a powerful tool for testing of web-applications (which is your target). So, if you want to go for free tool, then Selenium is a very good option to with and it is very popular among QAs in automation field. You can extend its use for Mobile, Desktop, Performance testing with use of free extensions, plugins and frameworks. While for other tools as mentioned by you either you have to buy multiple tools or you have to buy extension of same tool.

Our organization too is using Selenium with a wide range of projects (Mobile and Web applications).

However, these are some limitations of Selenium (which are no blocker for you):

  1. Its and open source tool so in case of any technical issues you need to rely on the selenium community forums to get your issue resolved. While for paid tool you just need to raise ticket and will get solution. But, selenium has a vast community and user support.
  2. You need to know at least one of the supported language very well in order to automate your application successfully. With other tools like Test Complete and Test Studio (it have a power recorder) you can go with recording, but at some point you have to enter into coding.
  3. No inbuilt reporting capability so you need plugins like JUnit and TestNG for test reports. This is what other paid tools provide you i.e. in built good looking reports.
  4. Lot of challenges with IE browser.

Things which Selenium offers but others doesn't are:-

  1. No Fee i.e. Free Tool (A bigger factor)
  2. Support for multiple languages you can use Java, C#, Python and ROR etc. for creating test cases.
  3. Lots of Testing frameworks to choose from (others have only one pre-defined framework to be used).
  4. Execute tests in parallel on multiple OS, browser combinations using Selenium Grid
  5. Supports all the popular browsers and versions (others too support multiple browsers)
  6. Integrate with DEV and CI environment seamlessly
  7. Most important: Already grown and more growing use of Selenium in automated testing, that almost every QA opening mention that person should know about Selenium. So, a good value to your career, skill set and resume.

Courtesy to following links:-

http://www.softwaretestingmentor.com/selenium-tutorials/limitations-of-selenium/ http://www.gallop.net/blog/tag/test-automation-tools-comparison/

If you compare selenium with the mentioned tools, you will find that every tool development company has already compared and published that analysis for people e.g. this link and have shown their tool is much better ;-) (I call it a sales strategy)

2
  • Thanks for that, as you say, its fairly easy to find what a commercial product claims to offer that Selenium Doesn't. But I have found finding the inverse is difficult. So making a business case based on what selenium offers that other providers Don't is proving quite difficult.
    – ECiurleo
    Oct 29, 2015 at 16:41
  • I have updated my answer for the required information too :)
    – Dhiman
    Oct 29, 2015 at 17:48
3

To add to @Dhiman's excellent answer, and noting that limits mentioned by @log_file was not a problem for me:

  • Selenium is W3C standard for browser automation, so it is here to stay.
  • For competent programmer, programming (instead of session recording) is not a bug, but a feature.

With competent programming team, you can develop pageobjects and reuse them in extremely flexible ways. We have multiple ways to generate data to feed into our integration testing system:

  • one of the paths collect some transaction data from production and replays them in testing system.
  • Another use of Selenium is to generate reports for the same datapoints in production and testing, and compare them to detect the differences. Because we replay data, we can analyze detected differences and automate major part of the diff analysis.
  • and of course standard regression tests.

So in some sense, some of our tests are data-driven (we can generate data sets for test, instead of endlessly replaying the same tests). Fuzzying, border conditions testing - just generate proper data sets.

Also in our experience, record/replay test are very fragile and sensitive to UI changes. After even a small change, many tests needs to be updated. With pageobject design pattern, such changes are localized and much simpler to manage.

I do not have experience using other competing automation framework (beyond dabbing into our legacy FitNesse a bit). But our company policy is to use open source tools for production whenever possible, because in our experience the response from commercial support is often delayed way into the future, and often forces you to "upgrade treadmill" - solution is "yes, we will fix it in next version of our library", so skipping version upgrades (as we like to do) would not be possible. From our POV such approach is unreliable - they are solving their problem, not ours.

With open source systems, if we do have some problem, usually someone else already solved it, and solution is discoverable by google search and some own development. Or you can always try to solve it yourself or hire an expert on hourly rate (buy time with money), which is not possible with closed systems.

With 2 years of experience in Selenium, I would not consider switching to proprietary test system if I can avoid it.

Yes, cross-browser testing is a problem, and IE is as usually the biggest offender - because MSFTies think that being incompatible with web standards will allow them to hang on the chunk of the market they have. If your IE customers are minority, incompatibilities are MSFT's problem not yours.

1

It really depends on the exact requirements your application has. We have been in a similar situation some time ago and we did a lot of research on this topic.

We have found a nice comparison of Ranorex vs. Selenium which helped us in the decision-making process.

However, in the end it is the best choice to give the tools a try and test them by yourself.

1
  • 2
    Please write the core points of the comparison here. The link may get broken
    – bish
    Dec 12, 2015 at 18:56
0

First things first- The question title and description don't match each other. You're asking features of Selenium that are not present in other tools, while in body you're asking the limitations. That's a dampner :-(

Back to the question- Dhiman has an excellent answer and he has covered almost all the points. Selenium as a functional web-testing tool is intensely powerful.

You can use the power of waits to handle AJAX calls, while you can use the power of various classes present inside the Selenium API to interact with dynamic web elements like Pop-ups, Modal windows, Drag-Drop Scenarios, Hover-over Menus etc.

Since you're talking about limitations, let me put forward some of the limitations I have personally felt, using Selenium over the last one and a half years :

  1. No Support for Captcha/Barcode Readers : Well that's the whole point of Captcha- so that it can't be automated and read by machines. If you want to automate this, you need to use external API's (and I'm not sure if they are successful).

  2. Selenium can't use proxy to connect to other websites: That's because Selenium, technically is already a proxy, and you can't really configure it to go through another proxy (at least not that I know of).

  3. Cross Browser Issues : It is true that Selenium gives you the flexibility to run your tests across browsers. It is also good at that. But since there are browser specific binding, you can't really be sure if a method is going to perform same way on Chrome, the same way it performs on Firefox. A good example is the Screenshot conundrum.

  4. Issues and Limitations while testing with Flash : I know that now a days Flash is used less in web applications. However, in case you website has any, you will find it hard to automate with Selenium. Here is a link on SO that discusses this.

  5. Problems in automating Java applets : You can read more about this on this SO link.

Points 4 and 5 are only situation specific, so they don't come into picture unless you have these in your web application.

4
  • Re "Screenshot conundrum" - seems that according to W3C it is not a problem at all - Chromium is doing the right think, and Firefox volunteered to do add to screenshot more than W3C required. Am I missing something? BTW +1 for limitations which are not issue for our usage. Nov 2, 2015 at 22:37
  • Well I may actually re-phrase it to not a limitation but an rather a challenge. The W3C specs are fine- but when you are designing tests- these things pose a challenge and IMHO should be universally same across all browser bindings. Nov 3, 2015 at 6:12
  • Yes they should - so Firefox should undo custom additions. :-) Browsers wars are over, for now. Nov 3, 2015 at 14:28
  • Thanks for the comment but I am not sure your assertions are correct; 1. there are ways round this, but its not functionality that any framework can offer as standard 2. Proxys are possible - docs.seleniumhq.org/docs/… 3. Browsers behave differently, if you need to take that into account your tests need to be write to reflect that.
    – ECiurleo
    Nov 3, 2015 at 16:15
-1

Selenium is at present the most intense Open Source Automation device accessible. It depends on java scripting to a huge degree. It is more suited for the new methodology like agile of coding and testing.

Supports languages like Java,Python,Perl,Ruby,c# and php etc. but QTP supports only vbscript.

It has support for all of the popular browsers like Internet Explorer, Firefox, Mozilla, Safari etc.

Selenium IDE is useful for paths.

Selenium IDE is Freeware.

Selenium IDE Easy to install.

Selenium IDE is the only tool of selenium that allows us to record user actions on browser window.

Run multiple tests at a time.

1
  • 1
    This was all already mentioned. Furthermore where are the sourced that Selenium is "the only tool for xxx"? Can you proof that there are no others?
    – bish
    Dec 2, 2015 at 6:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.