What automation tools to use with PHP development?

Question

I've used Selenium IDE to test Drupal (PHP) sites but want to create more advanced tests with loops and such. I hear Selenium WebDriver is all the rage, but when I google it, all these terms pop up and make me confused:

Selenium 2 API
(Google) WebDriver
Selenium WebDriver
Selenium Grid
Selenese
PHPUnit
NGUnit
HtmlUnit
HttpUnit
JUnit
Sikuli Script
SimpleTest
Watir
soapUI
Groovy

I'm wondering about how all these fit together in the grand scheme of unit testing, and, more specifially, Selenium WebDriver. Like, what's the difference between them, which are comparable (do about the same thing), where do they overlap, what can x do that y can't,

1. Can I use all of them to test PHP sites?
2. Which could I (and should I) use in conjunction (and why) and
3. Would I need additional software to make that work (if so, what software),
4. When should I use what,
5. Is Selenese something I use instead of PHP when coding tests and if so, what's the advantage of that
6. Are there any other tools that I should use in conjunction with Selenium Webdriver?

Sorry if the question is too broad or to nooby (please critizise constructively and give me a chance to rephrase my question before handing out too many minus points---this is my first question in this forum). Of course I'm not expecting anyone to answer all of this, but please chip in if you think you can clear my confusion in any way. By the way, my programming skills are at an entry level so please keep that in mind when explaining.

Answer 1

"How to eat an elephant? One bite at a time." --- The scope of your question is so wide, that it is not possible to shows all subtle differences between different technologies you list. It would be more useful for you to focus on a single problem (e.g., interaction with Web browser) and then search for technology that helps you solve this problem. Therefore, I decided to center my answer around generic problems the listed technologies try to solve.

I posted my answer as community wiki, so others can improve it. Many of the technologies you listed I know only superficially, so I hope others will add more here.

Unit testing

Test framework enables execution of test cases automatically and reporting test results. Test framework usually provides a set of assertion predicates (e.g., isEqual) that will be used to compare actual system behavior with expected behavior (output) you define. Having tests cases with similar setup you may group them in test suites.

The family of frameworks for unit testing, i.e. testing of single components and functions, is known as xUnit. JUnit and NGUnit are implementations of such framework of Java language. phpUnit and SimpleTest would be for unit testing of PHP code pieces.

Interacting with external systems

During test execution, a test case implementation passes input data to the system under test and gets actual output data in return. Systems under tests cannot be called directly, when they expose different interfaces (e.g., SOAP). Hence, to speak to such a system, test case must use "glue", i.e., test driver that uses the protocol understood by the system. A test driver make easier for you to interact with such systems: prepare input data, parse output and make assertions on them.

• SOAPUI: Talking to Web services (exposing SOAP or RESTful interface), Web resources (HTTP interface), JMS queues, backend services for Flash/Flex applications (AMF interface).
• HTTPUnit, HTMLUnit: Interacting at low level with HTML Web sites and Web resources with HTTP interface
• Selenium WebDriver, Google WebDriver: Talking to Web browser (IE, FF, Chrome).

Integration and system testing

Some of those test drivers, like HTTPUnit and WebDrivers, can be used together with xUnit frameworks to automate completely testing process. Yes, xUnit frameworks were originally designed for unit testing, but currently are often used for launching integration and system tests.

Programming Languages

Groovy is a programming language for the Java VM. If you write APIs in Groovy, you might choose to test those APIs in Groovy too. You might also choose to test Java APIs in Groovy, but there are other JVM-based languages that you could use to test Java APIs as well.

To quote its Wikipedia page, "Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots) of the software under test" and "Sikuli Script is a visual scripting API using Jython to create automation scripts."

Selenese is a set of commands for writing Selenium tests in tabular form. You can read more about it in this StackOverflow post.

Answer 2

Let me try to explain this in simple words so that we don't get confused with different jargon used with Selenium.

• Selenium 2 API, (Google) WebDriver, Selenium WebDriver are same things. Why we call it Selenium WebDriver because it is combination of Selenium (from ThoughtWorks) and WebDriver (from Google). We call it Selenium 2 also since WebDriver provides backward compatibility to Selenium 1. So if you want to start with automation, better to start with WebDriver.

• Selenium Grid is used for running automation in parallel mode. Suppose, we want to run a test on 3 browsers; IE, FF, Chrome, we use Grid.

• Selenium/Webdriver supports different testing frameworks (FW). These are different Unit Testing frameworks Selenese PHPUnit, NGUnit, HtmlUnit, HttpUnit, JUnit. Depending on the application, different FW have different benefits. We should thorough study our application and then choose FW appropriately.

• Selenium/Webdriver also supports different languages. So we can choose any language to write our script.

My personal opinion is: Selenium/Webdriver is best fit with Java language as it is written in Java and with we can use testing framework Junit or testNG.

Answer 3

I'm wondering about how all these fit together in the grand scheme of unit testing, and, more specifically, Selenium WebDriver.

Webdriver is used for end to end, user acceptance testing, running in a real browser, automation. This is usually not unit or white box testing, it is black box testing.

It is used with other technologies and approaches such as

• Selenium 2 API
• (Google) WebDriver
• Selenium WebDriver
• Selenium Grid
• Selenese
• Watir

On exception to this is a viual based tool for browser automation:

• Sikuli Script

Unit testing on the other hand is not about browser automation (though it can be about unit testing javascript). Products in this category:

• JUnit
• PHPunit
• NGUnit
• HtmlUnit (headless browser)
• httpUnit (headless browser)
• SimpleTest
• soapUI

Other:

• Groovy - a language for the java platform

So for a PHP developer I would look at PHPunit for unit tests and then one of the selenium based products. Usually you pick one of the main language bindings - java, python, c#, ruby, javascript, etc. to go with it.
For php I would also explore Behat and Mink as detailed at http://www.seleniumhq.org/about/platforms.jsp#programming-languages

Answer 4

You can conduct visual and usability testing manually, but for functionality, acceptance and unit testing, cross-browser testing, an automated PHP framework can help speed the test cycles substantially. So here are some of the best PHP frameworks that I have come across. I have also provided their pros and cons to help you decide better (as mentioned in Best PHP automated frameworks)

PHPUnit

---

PHPUnit is considered one of the best PHP frameworks for test automation, meant for unit testing of an application developed using PHP language. Just like JUnit, it is an instance of xUnit and works almost in a similar manner.

Pros:

1. Just like other test automation frameworks meant for unit testing, PHPUnit helps you in developing a code that performs well and is easy to maintain.
2. It also helps you to identify defects that may arise before the code is pushed to further testing phases.
3. Issues are detected early during the development phase since the testing is carried out by developers only.
4. Unit testing helps in detecting issues and fixing the code at a certain fragment of the application, thereby leaving other fragments intact and without any chance of breakage.
5. Debugging process is made simpler. Debugging is required only when a the certain unit test fails.

Cons: The only disadvantage of PHPUnit is that, for testing multiple functions, the developer is required to add cover annotations. By any chance, if you change the name of the method or function without updating the @covers annotation, testing is skipped for that certain method or function.

Codeception

---

Codeception is an automation testing framework having multiple features apart from only Unit testing. The framework is derived from PHPUnit and has the ability to manage Functional, Unit as well as Acceptance testing of a web application.

Pros:

1. First of all, you will get the best way to write test cases in PHP language. Syntax highlighting also eliminates the chance of mistakes in syntax.
2. Writing the test cases are very simple.
3. You can do not only unit testing but also others like Acceptance or functional testing just by adding suites.
4. Supports multiple frameworks without any need of an extension or changing the code. If your project is migrating, your old test cases written using Codeception will work just the same way.
5. The Db module of Codeception helps the tester to fetch whatever you need from the database and clean up the data from your test module. Isolating the tests thus becomes much easier.
6. You can run complex test cases by using XPath locators or CSS selectors. REST and SOAP web services can also be tested simply using Codeception.
7. The tests are simple and readable enough to be clearly understood by a person coming from a non-technical background.

Cons:

1. Since the framework emulates a browser, the chances of getting a false-positive result increase.
2. Requires a framework to start testing.
3. Cannot test AJAX and JavaScript.

Laravel Dusk

---

Laravel task is a browser-based test automation tool for testing PHP based web applications. This tool also helps you to automate repetitive tasks. Using this framework, you can either test the applications you developed or any other published website using Google Chrome.

Pros:

1. You can develop the best authentication and authorization script.
2. Presentation views like HTML and business logic like SQL queries can be easily separated.
3. The migration feature of Laravel allows you to create database tables of standard format. For developers, Laravel can be used with Angular JS for developing large single-page web applications.
4. The tool can also be used to create dynamic HTML elements since it supports Vue js as well.
5. Asset compilation can also be done with elixir or gulp that minifies js and CSS files, resulting in the faster performance of the application.

Cons:

The only disadvantage of Laravel is that it cannot be used on other browsers except Chrome. Nowadays when customers demand a website that is cross-browser compatible, this is certainly a drawback.

Behat

---

Behat is a behavior-driven testing tool for applications developed using PHP. The aim of the tool is to aid the nonverbal communication between stakeholders, management, and developers during the software development life cycle.

Pros:

1. Test cases are written in a human-readable manner, the features describe the specs of individual sites and the test suite provides very clear documentation consisting of the web application’s features.
2. Helps in maintaining an application having a stable deployment cycle. All you need to do is run the tests, detect and fix the bugs and repeat the process until the project passes all the test cases, resulting in a stable application.
3. The acceptance criteria being easy to understand, developers get clarity regarding what the customer wants. Without any blockage, they can start developing and add new functionality. Once the functionality successfully passes the test cases, it gets ready for deployment.
4. It helps you to work closely with the customer until you two mutually agree on how a product should work. Thereby, forcing you to follow the standard protocols of behavior-driven development.

Cons:

1. Behat takes quite some time for implementation.
2. You will also have to rethink and restructure the entire development process.
3. Discussion with stakeholders and developers may extend a lot until you come to a mutual understanding of how a feature will be developed.

PHPSpec

---

PHPSpec is a popular BDD tool which helps you to write clean code using PHP. The development works by first describing how the object behaves. Once that is approved by the stakeholders, the next phase is to start coding.

Pros:

1. While starting test case execution, PHPSpec generates skeletons of classes or methods based on the usage of the methods inside the test cases. Any additional edge that you face can be covered by writing an additional test case.
2. By refactoring, you can restructure the entire code without changing how it behaves externally.
3. You can also use PHPSpec to test applications developed using Laravel.
4. PHPSpec has a mocking framework known as prophecy. Prophecy ensures that for every test, the required class’s mock instance is passed to the constructor.

Cons:

1. Testing private methods is a limitation.
2. Let’s suppose your public method returns a value. It is not a part of your object API. You cannot test that.
3. Integration testing cannot be performed using PHPSpec.
4. No Matter how useful Prophecy is, it cannot mock abstract methods.

Atoum

---

Atoum is an automated framework meant for unit testing of an application that is developed using PHP language. Although it behaves a bit like PHPUnit, it is much more modern, simple and a standalone framework. Unlike PHPUnit, the test cases written using Atoum are easily readable and much simplified. Also, the asserters of Atoum enables you to write less code and test more functionalities.

Pros:

1. It has a flexible structure. It allows you to rename directory, test cases or suites and whatever you like.
2. There are 3 engines for executing the test cases. For a single test case, you can define a specific engine apart from the engine provided by default.
3. You can write a test case in multiple ways. Either by using the classic way, by an academic way using empty asserters or a much better and smarter way by using the asserters of Atoum.
4. Test cases are readable and easier to understand since Atoum provides a set of natural assertions.
5. You are no longer required to manipulate real directories. While running the test cases, Atoum allows you to create a virtual file system consisting of fake directories, reducing the risk of mishandling real project files.
6. Class constructors can be mocked. Whenever a method is called, you can control the computational value.
7. The best feature is, Atoum can easily integrate with continuous integration tools like Gitlab, Jenkins, PHPCI, Travis CI, etc. It can also be used with multiple IDEs like Atom, Netbeans, Sublime Text, etc.

Cons: I haven’t found any which cannot be resolved or creates a roadblock.

Kahlan

---

Kahlan is another behavior-driven test framework. What makes it different is that it takes BDD testing on PHP a step ahead by use of describe-it syntax. Just like Javascript or Ruby, using this framework, without any PECL extensions, you can monkey patch or stub your code directly.

Pros:

1. Inclusion of JSpec, RSpec syntax.
2. A much better code coverage metrics.
3. You can check whether a specific method is called on your class.
4. Stubbing system is much more flexible. Prophecy or Mockery is no longer required.
5. You can replace the core classes and functions easily by monkey patching.
6. Icov and Istanbul are built in reporters that allow easy HTML or terminal reporting.
7. Coveralls, Clobber, Climate are some built in exporters that allow you to easily push the code.

Cons: Kahlan does not have the ability of Storytelling or developing scenario based BDD.

Peridot

---

Peridot is a highly reliable, extensible PHP framework that supports testing of an application developed under behavior driven environment. The framework is lightweight and faster than PHPSpec or other frameworks. Also, the fact that it is event-driven makes it popular since developing plugins are much easier when you are using Peridot.

Pros:

1. Just like Kahlan, it also uses describe-it syntax to write natural and human readable test cases. Test cases written using Peridot can be easily understood by someone belonging to a nontechnical background.
2. Provides a painless experience of writing ad hocs since it is event-driven.
3. You can also add custom reporters, helpers and plugins easily.
4. If your test cases are bulky, Peridot can run them concurrently since it is lightweight and faster than other frameworks.

Cons: The only drawback is that it is not suitable to be used in an Agile application since its features are not vast like Behat.

Selenium

---

Selenium is the best automation framework for automated cross browser testing of a web application. For an application developed using PHP, you can use PHP bindings for developer browser-based automation test cases. The framework is flexible and the latest version of Selenium web driver runs flawlessly across all the major browsers.

Pros:

1. First of all the framework is open source. Anyone can download the source code and modify it according to project requirements.
2. It supports multiple programming languages. Even if your project someday migrates from PHP to any other technology, you can still use Selenium for executing the test cases.
3. Selenium can be used on any operating system and it supports all the major browsers like Chrome, IE, Opera, Safari or Firefox.
4. By using very less hardware resources, you can execute multiple test cases parallely using Selenium.

Cons:

1. The framework is a bit difficult and test case creation takes a bit of time. The learning curve is also a bit steep since the framework is evolving day by day.

2. Since the software is open-source, there is no reliable support if you face any complications.

3. Setting up the test environment is also complicated and also, it does not have any support for integration with test management tools in case you wish to report bugs.

4. You can only perform browser compatibility testing on the browsers available on your local machine.

I hope the tools mentioned above will help you choose the best PHP automated frameworks according to your project requirement.