TestNG vs Robot Framework - pros and cons

Question

What are the pros and cons of using Robot Framework with selenium2Library (webdriver).

Which one is the better option for web application testing - TestNG or Robot Framework?

Answer 1

TestNG has been created as an alternative to JUnit in unit testing. While, RobotFramework has been created for acceptance testing.

Each has remarkable features:

TestNG

• TestNG is easily integrated into maven cycle and used mostly with Data Driven testing
• TestNG is pure Java
• Annotations are easier to understand
• Test cases can be grouped more easily
• Parallel testing is possible
• TestNG supports only Java (so this is one of the limitation)

Robot Framework

• Robot Framework allows Keyword Driven Testing and even behavior driven one.
• Robot Framework was initially written in Python but latest version has Java bindings as well. Robot is highly extensible, in python, java, and/or any .NET language.
• Robot Framework is a generic, application and technology independent framework.
• Robot can be used for REST and SOAP service tests, database tests

Both have great reporting frameworks and both can be attached to various Continuous Integration like Jenkins.

And, both have Selenium support.

You should decide that which features you require for your testing and then use the same.

Answer 2

I think the question should be: Java WebDriver + TestNG Vs Robot.

I have worked on both Java + TestNG and RobotFramework. Here is my analysis by points.

Ease of use

• TestNG: You need to know Java & ANT/Maven, loggers and more libraries (depending on your need).

• Robot: Basic programming language is enough in any language. You can train your manual testing team.

Winner: Robot

Code writing

• TestNG: Basically, it is Java. Takes a bit more time compared to Robot. You need to write some extra lines which have nothing to do with your business. i.e. the definition of classes, scoping etc. However, people expert in writing code may argue here.

• Robot: Write only as much as you need for your automation. (Standard login page code will not be more than 6 lines)

Winner: Robot

Parallel Execution

• TestNG: Possible. You can do that with Selenium Grid or on Standalone machine

• Robot: Possible (using pabot). You can use selenium grid too.

Winner: Tie

Logging and Failure Analysis

• TestNG: If you are used to Java then it is easy. However, you don't get the logs as good/cool as Robot. I guess screenshot is not captured by default unless you implement some logic. you need to use Log4j for detailed logging.

• Robot: Neat and clean logs and reports. Comes with a screenshot.

Winner: Robot

Flexibility

• TestNG: This is Java. The Sky is the limit.

• Robot: There are many things you can't do. if-else (only if block). No nested loops. And many things if you need complex coding. For many things, you can write a custom keyword in Python.

Winner: TestNG (Java)

Framework Design

• TestNG: You have to design the framework.

• Robot: It's a readily available framework

Winner: Robot

Resource Availablity

• TestNG/Java: It is so easy to find a person who knows Java/TestNG. If you don't know Java, there are plenty of developer in dev team who can help the manual tester to begin. However, quality of code will vary depending on who is writing code.

• RobotFramework: It is hard to find someone who has worked on RF. Though, the learning curve is very short compared to Java. All one need to know any programming knowledge. Basic knowledge of Python is needed if someone wants to write a customized library.

Winner: Java

Conclusion

In most of the cases, I feel Robot will do whatever you want to do. It's supportive libraries are increasing with time. You can do Web, API, Mobile, SSH, DB and many types of automation easily without knowing much in details.

Answer 3

1. Robot is keywords driven(mostly) framework, where testng is data driven Framework

2. You need to design complete framework on you own for the testng , where in the RF you get pretty much from robot framework itself.

3. Writing test case are much more easy in the RF compared to the testNG.

4.for completed test, user can design their own customised keywords in the RF.

5. RF can support any language, where for testNG it is little difficult to write code in other languages rather than java.

6.reporting is much much easier in RF , in the testNG user need to write own code for proper reporting of the result.

7.testNG needed lot of 3 party tools to actually create, run and to generate clean report compared to RF

Answer 4

In addition to great answers I would like to mention some disadvantages of Robot Framework (RF).

Robot Framework provides external DSL (Domain Specific Language). As such it suffers from limitations that external DSLs have:

have no mother language to depend upon, and so they must reinvent much general purpose behavior.

This means that many features that generic-purpose languages, like Python or Java, have been having for years, in Robot Framework do not exist or look immature. As relatively your language, it promotes many bad practices such as global variables. Global variables are bad because:

since every function has access to these, it becomes increasingly hard to figure out which functions actually read and write these variables.

You won't see this as a problem is a small project, but as your test base starts growing it will become hard for you to understand and refactor the code.

Here we come to the second limitation. RF does not have support from popular IDEs like Intellij, which makes it hard to debug and refactor tests. This becomes a nightmare when you want to refactor Python functions or parameters already used by your RF tests. You will need to refactor them manually both in RF tests and in Python fixtures.

Robot Framework is also overly verbose and thus hard to read. Here's an example of a simple loop:

@{ITEMS}    Create List    Star Trek    Star Wars    Perry Rhodan
:FOR    ${ELEMENT}    IN    @{ITEMS}
\    Log    ${ELEMENT}
\    ${ELEMENT}    Replace String    ${ELEMENT}    ${SPACE}    ${EMPTY}
\    Log    ${ELEMENT}

In Python that would be only:

items = ['Star Trek', 'Star Wars', 'Perry Rhodan']
for element in items:
  element = element.replace(' ', '')
  log(element)

Clearer, isn't it? RF abuses upper case in naming variables which makes text hard to read.

The syntax, external DSL of RF is constraining. However, other features of RF like logging, large number of fixtures (Selenium integration, REST API clients, etc) can be useful. And you still can you those fixtures without the burden of RF syntax, because they are written in Python. Just write your tests in Python.