7

What are the pros and cons of using Robot Framework with selenium2Library (webdriver).

Which one is the better option for web application testing - TestNG or Robot Framework?

13

TestNG has been created as an alternative to JUnit in unit testing. While, RobotFramework has been created for acceptance testing.

Each has remarkable features:

TestNG

  • TestNG is easily integrated into maven cycle and used mostly with Data Driven testing
  • TestNG is pure Java
  • Annotations are easier to understand
  • Test cases can be grouped more easily
  • Parallel testing is possible
  • TestNG supports only Java (so this is one of the limitation)

Robot Framework

  • Robot Framework allows Keyword Driven Testing and even behavior driven one.
  • Robot Framework was initially written in Python but latest version has Java bindings as well. Robot is highly extensible, in python, java, and/or any .NET language.
  • Robot Framework is a generic, application and technology independent framework.
  • Robot can be used for REST and SOAP service tests, database tests

Both have great reporting frameworks and both can be attached to various Continuous Integration like Jenkins.

And, both have Selenium support.

You should decide that which features you require for your testing and then use the same.

  • I would add RF has no support for refactoring and debugging in popular IDEs like Intellij – dzieciou Jun 13 at 18:20
  • Also, as an external DSL it has no mother language to depend upon, and so it must reinvent much general purpose behavior like loops, conditional clauses, etc.: modeling-languages.com/internal-external-dsls – dzieciou Jun 13 at 18:21
7

I think the question should be: Java WebDriver + TestNG Vs Robot.

I have worked on both Java + TestNG and RobotFramework. Here is my analysis by points.

Ease of use

  • TestNG: You need to know Java & ANT/Maven, loggers and more libraries (depending on your need).

  • Robot: Basic programming language is enough in any language. You can train your manual testing team.

Winner: Robot

Code writing

  • TestNG: Basically, it is Java. Takes a bit more time compared to Robot. You need to write some extra lines which have nothing to do with your business. i.e. the definition of classes, scoping etc. However, people expert in writing code may argue here.

  • Robot: Write only as much as you need for your automation. (Standard login page code will not be more than 6 lines)

Winner: Robot

Parallel Execution

  • TestNG: Possible. You can do that with Selenium Grid or on Standalone machine

  • Robot: Possible (using pabot). You can use selenium grid too.

Winner: Tie

Logging and Failure Analysis

  • TestNG: If you are used to Java then it is easy. However, you don't get the logs as good/cool as Robot. I guess screenshot is not captured by default unless you implement some logic. you need to use Log4j for detailed logging.

  • Robot: Neat and clean logs and reports. Comes with a screenshot.

Winner: Robot

Flexibility

  • TestNG: This is Java. The Sky is the limit.

  • Robot: There are many things you can't do. if-else (only if block). No nested loops. And many things if you need complex coding. For many things, you can write a custom keyword in Python.

Winner: TestNG (Java)

Framework Design

  • TestNG: You have to design the framework.

  • Robot: It's a readily available framework

Winner: Robot

Resource Availablity

  • TestNG/Java: It is so easy to find a person who knows Java/TestNG. If you don't know Java, there are plenty of developer in dev team who can help the manual tester to begin. However, quality of code will vary depending on who is writing code.

  • RobotFramework: It is hard to find someone who has worked on RF. Though, the learning curve is very short compared to Java. All one need to know any programming knowledge. Basic knowledge of Python is needed if someone wants to write a customized library.

Winner: Java

Conclusion

In most of the cases, I feel Robot will do whatever you want to do. It's supportive libraries are increasing with time. You can do Web, API, Mobile, SSH, DB and many types of automation easily without knowing much in details.

0
  1. Robot is keywords driven(mostly) framework, where testng is data driven Framework

  2. You need to design complete framework on you own for the testng , where in the RF you get pretty much from robot framework itself.

  3. Writing test case are much more easy in the RF compared to the testNG.

4.for completed test, user can design their own customised keywords in the RF.

  1. RF can support any language, where for testNG it is little difficult to write code in other languages rather than java.

6.reporting is much much easier in RF , in the testNG user need to write own code for proper reporting of the result.

7.testNG needed lot of 3 party tools to actually create, run and to generate clean report compared to RF

0

In addition to great answers I would like to mention some disadvantages of Robot Framework (RF).

Robot Framework provides external DSL (Domain Specific Language). As such it suffers from limitations that external DSLs have:

have no mother language to depend upon, and so they must reinvent much general purpose behavior.

This means that many features that generic-purpose languages, like Python or Java, have been having for years, in Robot Framework do not exist or look immature. As relatively your language, it promotes many bad practices such as global variables. Global variables are bad because:

since every function has access to these, it becomes increasingly hard to figure out which functions actually read and write these variables.

You won't see this as a problem is a small project, but as your test base starts growing it will become hard for you to understand and refactor the code.

Here we come to the second limitation. RF does not have support from popular IDEs like Intellij, which makes it hard to debug and refactor tests. This becomes a nightmare when you want to refactor Python functions or parameters already used by your RF tests. You will need to refactor them manually both in RF tests and in Python fixtures.

Robot Framework is also overly verbose and thus hard to read. Here's an example of a simple loop:

@{ITEMS}    Create List    Star Trek    Star Wars    Perry Rhodan
:FOR    ${ELEMENT}    IN    @{ITEMS}
\    Log    ${ELEMENT}
\    ${ELEMENT}    Replace String    ${ELEMENT}    ${SPACE}    ${EMPTY}
\    Log    ${ELEMENT}

In Python that would be only:

items = ['Star Trek', 'Star Wars', 'Perry Rhodan']
for element in items:
  element = element.replace(' ', '')
  log(element)

Clearer, isn't it? RF abuses upper case in naming variables which makes text hard to read.

The syntax, external DSL of RF is constraining. However, other features of RF like logging, large number of fixtures (Selenium integration, REST API clients, etc) can be useful. And you still can you those fixtures without the burden of RF syntax, because they are written in Python. Just write your tests in Python.

  • 1
    Loops and such are really the poor part of RF but ... One can easily just move the bad parts into python side or just use eval inline. There's also a work ongoing (for rf 4.0 afaik) to implement proper parser which should bring help to language constructs And third, you example about RF code uses very old syntax for loops. – rasjani Jun 28 at 9:59
  • @rasjani Agreed. I have focused mainly on the bad part of RF. Sure one can move that part to Python but then why introduce it to Robot Framework at all? This only encouraged bad practices. – dzieciou Jun 28 at 10:08
  • @rasjani Using eval inline is... a bad practice IMHO. It mixing two languages and abstractions in one document and will make reading the tests harder. – dzieciou Jun 28 at 10:24
  • so essentially your logic would also mean that people shouldnt be using any keyword libraries because they are typically written in python and thus is this encourages bad practices ? There's a saying about only having a hammer that fits here. Some scenarios just make much more sense to implement in "real" programming languages and just expose them as generic keywords for readability and maintainability. About eval, it was just an example as where your snippets. Like in any code, there are always multiple ways to archive things and optimal way depends case by case. – rasjani Jun 28 at 10:27
  • For example, one could place the eval portions into a separate keyword/resource file and making it "reusable" and hidden from the actual tests. Thus, the comment I left was to just highlight that things are not that black and white as your comment paints it. Nothing personal. – rasjani Jun 28 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.