3

I've used Selenium IDE to test Drupal (PHP) sites but want to create more advanced tests with loops and such. I hear Selenium WebDriver is all the rage, but when I google it, all these terms pop up and make me confused:

Selenium 2 API
(Google) WebDriver
Selenium WebDriver
Selenium Grid
Selenese
PHPUnit
NGUnit
HtmlUnit
HttpUnit
JUnit
Sikuli Script
SimpleTest
Watir
soapUI
Groovy

I'm wondering about how all these fit together in the grand scheme of unit testing, and, more specifially, Selenium WebDriver. Like, what's the difference between them, which are comparable (do about the same thing), where do they overlap, what can x do that y can't,

  1. Can I use all of them to test PHP sites?
  2. Which could I (and should I) use in conjunction (and why) and
  3. Would I need additional software to make that work (if so, what software),
  4. When should I use what,
  5. Is Selenese something I use instead of PHP when coding tests and if so, what's the advantage of that
  6. Are there any other tools that I should use in conjunction with Selenium Webdriver?

Sorry if the question is too broad or to nooby (please critizise constructively and give me a chance to rephrase my question before handing out too many minus points---this is my first question in this forum). Of course I'm not expecting anyone to answer all of this, but please chip in if you think you can clear my confusion in any way. By the way, my programming skills are at an entry level so please keep that in mind when explaining.

4
  • You are correct, this is a broad question. More accurately, it could be interpreted in a broad way. JKS's approach of grouping them into categories helps to narrow the question significantly.
    – corsiKa
    Nov 16 '12 at 15:56
  • 3
    One other thing I would say is that the hallmark of a great question (as in, what separates a great question from a mediocre one) is exhibition of prior effort. So when you have the list of things that you don't know what they are, for example, a great question says "I saw all these things - here's what they appear to be from my reading about them" as well as any connections you draw from that. This helps anyone who answers understand more about what -you- understand, and avoids covering things you already know. Make sense?
    – corsiKa
    Nov 16 '12 at 15:57
  • 1
    Welcome to SQA, Johanna. I am downvoting this because of its breadth. It is a reasonable question for a beginner to ask, but in SQA we encourage questioners to be more specific and to present evidence that they have researched the question beforehand. As you learn more, please feel to follow up with additional (and more specific) questions.
    – user246
    Nov 16 '12 at 16:49
  • Thanks! You're of course right, I should have been more clear about what I already knew. I had a hard time knowing where to take my first bite of the elephant and went a little lazy. The answers I've gotten have been great. I'll have some more time to read and reflect tomorrow and will be sure to follow up.
    – Johanna
    Nov 17 '12 at 16:11
5

"How to eat an elephant? One bite at a time." --- The scope of your question is so wide, that it is not possible to shows all subtle differences between different technologies you list. It would be more useful for you to focus on a single problem (e.g., interaction with Web browser) and then search for technology that helps you solve this problem. Therefore, I decided to center my answer around generic problems the listed technologies try to solve.

I posted my answer as community wiki, so others can improve it. Many of the technologies you listed I know only superficially, so I hope others will add more here.

Unit testing

Test framework enables execution of test cases automatically and reporting test results. Test framework usually provides a set of assertion predicates (e.g., isEqual) that will be used to compare actual system behavior with expected behavior (output) you define. Having tests cases with similar setup you may group them in test suites.

The family of frameworks for unit testing, i.e. testing of single components and functions, is known as xUnit. JUnit and NGUnit are implementations of such framework of Java language. phpUnit and SimpleTest would be for unit testing of PHP code pieces.

Interacting with external systems

During test execution, a test case implementation passes input data to the system under test and gets actual output data in return. Systems under tests cannot be called directly, when they expose different interfaces (e.g., SOAP). Hence, to speak to such a system, test case must use "glue", i.e., test driver that uses the protocol understood by the system. A test driver make easier for you to interact with such systems: prepare input data, parse output and make assertions on them.

  • SOAPUI: Talking to Web services (exposing SOAP or RESTful interface), Web resources (HTTP interface), JMS queues, backend services for Flash/Flex applications (AMF interface).
  • HTTPUnit, HTMLUnit: Interacting at low level with HTML Web sites and Web resources with HTTP interface
  • Selenium WebDriver, Google WebDriver: Talking to Web browser (IE, FF, Chrome).

Integration and system testing

Some of those test drivers, like HTTPUnit and WebDrivers, can be used together with xUnit frameworks to automate completely testing process. Yes, xUnit frameworks were originally designed for unit testing, but currently are often used for launching integration and system tests.

Programming Languages

Groovy is a programming language for the Java VM. If you write APIs in Groovy, you might choose to test those APIs in Groovy too. You might also choose to test Java APIs in Groovy, but there are other JVM-based languages that you could use to test Java APIs as well.

To quote its Wikipedia page, "Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots) of the software under test" and "Sikuli Script is a visual scripting API using Jython to create automation scripts."

Selenese is a set of commands for writing Selenium tests in tabular form. You can read more about it in this StackOverflow post.

3
  • Many thanks @dzieciou for this great answer. The only big question mark I have left now is about SoapUI - could it be used in conjunction with Selenium WebDriver or Selenium WebDriver+xUnit and would there be an advantage to that? Or are they entirely separate things, not used together?
    – Johanna
    Nov 18 '12 at 23:04
  • Using SOAP UI and Selenium would make sense if you want to access both Web site (with Selenium) and Web service from the same xUnit test case, e.g. you want to check what changed to UI after calling Web service. To call Web service you use Web service client stub generated with SOAP UI. Another case is to organize tests together and run both Web site/Selenium test cases and SOAP UI cases from the same xUnit test suite.
    – dzieciou
    Nov 19 '12 at 6:47
  • What I haven't wrote is that SOAP UI GUI enables writing complete tests (invocations, assertions) for Web service. You may think of it like Selenium IDE for Web services, but instead of recording you define data flow and its conditions.
    – dzieciou
    Nov 19 '12 at 6:50
6

Let me try to explain this in simple words so that we don't get confused with different jargon used with Selenium.

  • Selenium 2 API, (Google) WebDriver, Selenium WebDriver are same things. Why we call it Selenium WebDriver because it is combination of Selenium (from ThoughtWorks) and WebDriver (from Google). We call it Selenium 2 also since WebDriver provides backward compatibility to Selenium 1. So if you want to start with automation, better to start with WebDriver.

  • Selenium Grid is used for running automation in parallel mode. Suppose, we want to run a test on 3 browsers; IE, FF, Chrome, we use Grid.

  • Selenium/Webdriver supports different testing frameworks (FW). These are different Unit Testing frameworks Selenese PHPUnit, NGUnit, HtmlUnit, HttpUnit, JUnit. Depending on the application, different FW have different benefits. We should thorough study our application and then choose FW appropriately.

  • Selenium/Webdriver also supports different languages. So we can choose any language to write our script.

My personal opinion is: Selenium/Webdriver is best fit with Java language as it is written in Java and with we can use testing framework Junit or testNG.

8
  • I would also add that Groovy isn't so much a testing tool but simply a java-esque scripting language. It's very easy for a Java programmer to pick it up!
    – corsiKa
    Nov 16 '12 at 15:29
  • Also, one minor thing would be that you'd use XUnit to test X language. So PHPUnit for php unit tests, JUnit for Java Unit tests, etc.
    – corsiKa
    Nov 16 '12 at 15:32
  • 1
    @corsiKa: Not necessarily. You may use JUnit+Selenium to test Web sites made in PHP, because you do not interact with PHP directly. I.e. you do not use JUnit for unit testing here.
    – dzieciou
    Nov 16 '12 at 15:34
  • @dzieciou One could also use Selenium to keep themselves logged in to StackOverflow to farm the Fanatic badge. That doesn't mean it's what it was designed to do.
    – corsiKa
    Nov 16 '12 at 15:58
  • @corsiKa: What's the actual disadvantage of using JUnit to test Web pages wrote in PHP? xUnit has "unit" in name but lot's of testers uses it for integration and system tests, e.g. combine JUnit with Fitnesse.
    – dzieciou
    Nov 16 '12 at 16:01
0

I'm wondering about how all these fit together in the grand scheme of unit testing, and, more specifically, Selenium WebDriver.

Webdriver is used for end to end, user acceptance testing, running in a real browser, automation. This is usually not unit or white box testing, it is black box testing.

It is used with other technologies and approaches such as

  • Selenium 2 API
  • (Google) WebDriver
  • Selenium WebDriver
  • Selenium Grid
  • Selenese
  • Watir

On exception to this is a viual based tool for browser automation:

  • Sikuli Script

Unit testing on the other hand is not about browser automation (though it can be about unit testing javascript). Products in this category:

  • JUnit
  • PHPunit
  • NGUnit
  • HtmlUnit (headless browser)
  • httpUnit (headless browser)
  • SimpleTest
  • soapUI

Other:

  • Groovy - a language for the java platform

So for a PHP developer I would look at PHPunit for unit tests and then one of the selenium based products. Usually you pick one of the main language bindings - java, python, c#, ruby, javascript, etc. to go with it.
For php I would also explore Behat and Mink as detailed at http://www.seleniumhq.org/about/platforms.jsp#programming-languages

0

You can conduct visual and usability testing manually, but for functionality, acceptance and unit testing, cross-browser testing, an automated PHP framework can help speed the test cycles substantially. So here are some of the best PHP frameworks that I have come across. I have also provided their pros and cons to help you decide better (as mentioned in Best PHP automated frameworks)

PHPUnit


PHPUnit is considered one of the best PHP frameworks for test automation, meant for unit testing of an application developed using PHP language. Just like JUnit, it is an instance of xUnit and works almost in a similar manner.

Pros:

  1. Just like other test automation frameworks meant for unit testing, PHPUnit helps you in developing a code that performs well and is easy to maintain.
  2. It also helps you to identify defects that may arise before the code is pushed to further testing phases.
  3. Issues are detected early during the development phase since the testing is carried out by developers only.
  4. Unit testing helps in detecting issues and fixing the code at a certain fragment of the application, thereby leaving other fragments intact and without any chance of breakage.
  5. Debugging process is made simpler. Debugging is required only when a the certain unit test fails.

Cons: The only disadvantage of PHPUnit is that, for testing multiple functions, the developer is required to add cover annotations. By any chance, if you change the name of the method or function without updating the @covers annotation, testing is skipped for that certain method or function.

Codeception


Codeception is an automation testing framework having multiple features apart from only Unit testing. The framework is derived from PHPUnit and has the ability to manage Functional, Unit as well as Acceptance testing of a web application.

Pros:

  1. First of all, you will get the best way to write test cases in PHP language. Syntax highlighting also eliminates the chance of mistakes in syntax.
  2. Writing the test cases are very simple.
  3. You can do not only unit testing but also others like Acceptance or functional testing just by adding suites.
  4. Supports multiple frameworks without any need of an extension or changing the code. If your project is migrating, your old test cases written using Codeception will work just the same way.
  5. The Db module of Codeception helps the tester to fetch whatever you need from the database and clean up the data from your test module. Isolating the tests thus becomes much easier.
  6. You can run complex test cases by using XPath locators or CSS selectors. REST and SOAP web services can also be tested simply using Codeception.
  7. The tests are simple and readable enough to be clearly understood by a person coming from a non-technical background.

Cons:

  1. Since the framework emulates a browser, the chances of getting a false-positive result increase.
  2. Requires a framework to start testing.
  3. Cannot test AJAX and JavaScript.

Laravel Dusk


Laravel task is a browser-based test automation tool for testing PHP based web applications. This tool also helps you to automate repetitive tasks. Using this framework, you can either test the applications you developed or any other published website using Google Chrome.

Pros:

  1. You can develop the best authentication and authorization script.
  2. Presentation views like HTML and business logic like SQL queries can be easily separated.
  3. The migration feature of Laravel allows you to create database tables of standard format. For developers, Laravel can be used with Angular JS for developing large single-page web applications.
  4. The tool can also be used to create dynamic HTML elements since it supports Vue js as well.
  5. Asset compilation can also be done with elixir or gulp that minifies js and CSS files, resulting in the faster performance of the application.

Cons:

The only disadvantage of Laravel is that it cannot be used on other browsers except Chrome. Nowadays when customers demand a website that is cross-browser compatible, this is certainly a drawback.

Behat


Behat is a behavior-driven testing tool for applications developed using PHP. The aim of the tool is to aid the nonverbal communication between stakeholders, management, and developers during the software development life cycle.

Pros:

  1. Test cases are written in a human-readable manner, the features describe the specs of individual sites and the test suite provides very clear documentation consisting of the web application’s features.
  2. Helps in maintaining an application having a stable deployment cycle. All you need to do is run the tests, detect and fix the bugs and repeat the process until the project passes all the test cases, resulting in a stable application.
  3. The acceptance criteria being easy to understand, developers get clarity regarding what the customer wants. Without any blockage, they can start developing and add new functionality. Once the functionality successfully passes the test cases, it gets ready for deployment.
  4. It helps you to work closely with the customer until you two mutually agree on how a product should work. Thereby, forcing you to follow the standard protocols of behavior-driven development.

Cons:

  1. Behat takes quite some time for implementation.
  2. You will also have to rethink and restructure the entire development process.
  3. Discussion with stakeholders and developers may extend a lot until you come to a mutual understanding of how a feature will be developed.

PHPSpec


PHPSpec is a popular BDD tool which helps you to write clean code using PHP. The development works by first describing how the object behaves. Once that is approved by the stakeholders, the next phase is to start coding.

Pros:

  1. While starting test case execution, PHPSpec generates skeletons of classes or methods based on the usage of the methods inside the test cases. Any additional edge that you face can be covered by writing an additional test case.
  2. By refactoring, you can restructure the entire code without changing how it behaves externally.
  3. You can also use PHPSpec to test applications developed using Laravel.
  4. PHPSpec has a mocking framework known as prophecy. Prophecy ensures that for every test, the required class’s mock instance is passed to the constructor.

Cons:

  1. Testing private methods is a limitation.
  2. Let’s suppose your public method returns a value. It is not a part of your object API. You cannot test that.
  3. Integration testing cannot be performed using PHPSpec.
  4. No Matter how useful Prophecy is, it cannot mock abstract methods.

Atoum


Atoum is an automated framework meant for unit testing of an application that is developed using PHP language. Although it behaves a bit like PHPUnit, it is much more modern, simple and a standalone framework. Unlike PHPUnit, the test cases written using Atoum are easily readable and much simplified. Also, the asserters of Atoum enables you to write less code and test more functionalities.

Pros:

  1. It has a flexible structure. It allows you to rename directory, test cases or suites and whatever you like.
  2. There are 3 engines for executing the test cases. For a single test case, you can define a specific engine apart from the engine provided by default.
  3. You can write a test case in multiple ways. Either by using the classic way, by an academic way using empty asserters or a much better and smarter way by using the asserters of Atoum.
  4. Test cases are readable and easier to understand since Atoum provides a set of natural assertions.
  5. You are no longer required to manipulate real directories. While running the test cases, Atoum allows you to create a virtual file system consisting of fake directories, reducing the risk of mishandling real project files.
  6. Class constructors can be mocked. Whenever a method is called, you can control the computational value.
  7. The best feature is, Atoum can easily integrate with continuous integration tools like Gitlab, Jenkins, PHPCI, Travis CI, etc. It can also be used with multiple IDEs like Atom, Netbeans, Sublime Text, etc.

Cons: I haven’t found any which cannot be resolved or creates a roadblock.

Kahlan


Kahlan is another behavior-driven test framework. What makes it different is that it takes BDD testing on PHP a step ahead by use of describe-it syntax. Just like Javascript or Ruby, using this framework, without any PECL extensions, you can monkey patch or stub your code directly.

Pros:

  1. Inclusion of JSpec, RSpec syntax.
  2. A much better code coverage metrics.
  3. You can check whether a specific method is called on your class.
  4. Stubbing system is much more flexible. Prophecy or Mockery is no longer required.
  5. You can replace the core classes and functions easily by monkey patching.
  6. Icov and Istanbul are built in reporters that allow easy HTML or terminal reporting.
  7. Coveralls, Clobber, Climate are some built in exporters that allow you to easily push the code.

Cons: Kahlan does not have the ability of Storytelling or developing scenario based BDD.

Peridot


Peridot is a highly reliable, extensible PHP framework that supports testing of an application developed under behavior driven environment. The framework is lightweight and faster than PHPSpec or other frameworks. Also, the fact that it is event-driven makes it popular since developing plugins are much easier when you are using Peridot.

Pros:

  1. Just like Kahlan, it also uses describe-it syntax to write natural and human readable test cases. Test cases written using Peridot can be easily understood by someone belonging to a nontechnical background.
  2. Provides a painless experience of writing ad hocs since it is event-driven.
  3. You can also add custom reporters, helpers and plugins easily.
  4. If your test cases are bulky, Peridot can run them concurrently since it is lightweight and faster than other frameworks.

Cons: The only drawback is that it is not suitable to be used in an Agile application since its features are not vast like Behat.

Selenium


Selenium is the best automation framework for automated cross browser testing of a web application. For an application developed using PHP, you can use PHP bindings for developer browser-based automation test cases. The framework is flexible and the latest version of Selenium web driver runs flawlessly across all the major browsers.

Pros:

  1. First of all the framework is open source. Anyone can download the source code and modify it according to project requirements.
  2. It supports multiple programming languages. Even if your project someday migrates from PHP to any other technology, you can still use Selenium for executing the test cases.
  3. Selenium can be used on any operating system and it supports all the major browsers like Chrome, IE, Opera, Safari or Firefox.
  4. By using very less hardware resources, you can execute multiple test cases parallely using Selenium.

Cons:

  1. The framework is a bit difficult and test case creation takes a bit of time. The learning curve is also a bit steep since the framework is evolving day by day.

  2. Since the software is open-source, there is no reliable support if you face any complications.

  3. Setting up the test environment is also complicated and also, it does not have any support for integration with test management tools in case you wish to report bugs.

  4. You can only perform browser compatibility testing on the browsers available on your local machine.

I hope the tools mentioned above will help you choose the best PHP automated frameworks according to your project requirement.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.