Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

We are choosing what system to start using in our company.

  1. it should be used for both backend (REST API, some DB checks) and UI testing
  2. it should use a simple language so even non-programmers/tester can understand the test cases (Product Owners should be able to see whether all acceptance criteria are covered)
  3. it should support integration with Jenkins
  4. it should support versioning of test cases so that for a particular product version we also can check out relevant test cases
  5. right now we use TestRail (test case management SW); so would be nice if it integrates with it (at least it is possible to program it so send test results there) or completely replaces it

Any ideas, experience?

Thanks

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Overview

Robot framework is an excellent choice that meets all of your goals. Robot can be used for UI tests (via selenium), REST and SOAP service tests, database tests, and just about any other type of acceptance test. You can even use robot tests to improve your manual testing process.

Robot is keyword driven

Robot is keyword-driven, which makes it very easy to create test cases that can be understood by product managers and stakeholders, without them needing to learn a programming language. Also because it is keyword driven, it makes it easier for QA professionals with limited technical expertise to write tests. What I like about robot over some similar tools such as cucumber is that you have the choice to write BDD-style tests (given/when/then), procedural tests, and data-driven tests. You aren't locked into a single testing strategy -- you can choose the style that best fits each scenario.

Robot is highly extensible

Robot is highly extensible, in python, java, and/or any .NET language. In fact, you can use just about any language at all through the remote API that robot has. The developers on your team can participate in testing by writing the more difficult keywords in a programming language while letting the QA people focus on test scenarios, coverage, etc.

Robot integrates with development tools

Because tests are plain text, they integrate extremely well with version control systems. They diff and merge nicely, especially if you use the plain-text, pipe-delimited format. There is a robot-specific jenkins plugin, and has a command line test runner, so it is extremely easy to integrate with Jenkins.

Also because the tests are plain text, your test writes (both developers and QA) can use whatever tool they are most comfortable with. Developers can use emacs, visual studio, eclipse, etc. non-developers can use brackets, notepad++, sublime text, etc. Many of these editors have robot-specific programming modes that provide syntax highlighting and other features. You aren't forced to use a specialized tool.

Robot has robust reporting

Robot generates a very simple-to-parse xml output file, and has an option to generate an xunit-style output file. This makes it pretty easy to integrate with other tools. It also has an interface that can call python functions for various test events -- for example, every time a suite finishes. Since testrail has a web-based API, it would be easy to set it up so that every time a suite finishes, it can send the results to testrail.

Other advantages

Some people prefer a real programming language when interacting with selenium, and that's certainly a great way to go if your team has the skill to do that. Most QA teams don't have that luxury. Even if you do, there are other advantages to using a framework like robot: built-in reporting, a fabulous tagging mechanism, and other features that just don't come with pure programming environments such as the robot listener interface.

Summary

At the time that I write this I've used robot framework at three different organizations. For one of those it wasn't an organizational fit and was eventually abandoned. Developers did a lot of the testing and they wanted a real programming language. In the other two, there were more dedicated QA who write tests, and for both of those organizations, robot worked extremely well and was embraced by the entire organization -- POs, developers and QA alike.

Is it the best tool? No, there is no best. It is, however, extremely flexible, extremely extensible, and is a very effective testing tool.

share|improve this answer
    
Nice explanation, thanks Bryan! After a short comparison we have chosen to start using Robot by both dev teams for acceptance module-level testing and also QAs for acceptance product-level (integrated modules) testing. –  Radek S Aug 10 at 15:53

Selenium/Python is what we use and are very happy with it.

  • Re #1: You can write tests in pure Python (without Selenium) to test rest/DB and run them using same test runner. Test are just python programs using Selenium libraries or whatever else you need. And there are libraries for anything I ever needed.
  • Re #2: Python is famously easy to read. Our own problem area experts can read code even if they cannot write it. All they took is free online Python course on Codecademy.
  • Re #3: yes.
  • Re #4: source code for test cases are in same subversion repository as the application, so they are in sync.
  • Re #5: No idea, sorry.

Edit: Ok I read about Robot in wikipedia and I can see why you are comparing to Fitnesse. We used Fitnesse before (and still maintain huge codebase of tests, which we plan to convert to plain Selenium in some distant future). Like Fitnesse, it uses tabular "language" to allow "non-programmer end users" to write tests. And IMHO in 10 years, like Fitnesse users (including us), Robot users will realize that non-programmer end users are absolutely not interested in writing tests - they prefer to hire programmers to do that. And programmers vastly prefer flexible universal programming language like Python, instead of some contraption with limited use, usability and extendability. Language which you extend any way your application requires, not limited by API for plugins or whatever you have. Language which has many tools and huge user community, instead of obscure niche. And this big community will steal any ideas worth stealing from other testing tools and integrate them to the tool they use.

Also, Selenium WebDriver is draft W3C standard for browser automation. Developed and used by Google, Facebook and Mozilla. So I feel pretty confident that after dust settles, this time I made a bet on right technology :-)

Of course you can find few people on Stack Overflow who like anything under the sun. Most Selenium users do not care for either Robot or Fitnesse - and like me, mostly they know why.

share|improve this answer
    
Thanks Peter. Actually for backend (REST API typically) we use Python. The reason I want to change it is because when I saw how much time it takes new testers (who claim they know Python somehow) to understand how to use the existing python framework/libs that we built, and then especially when they start testing, it fails and these people needs to check whether it is an issue in the tested system, "testing framework" or a test case, I was pretty frustrated. I like Python, but maybe it is too much freedom. –  Radek S Aug 4 at 11:17
    
BTW, I also asked at StackOverflow and people like Robot: stackoverflow.com/questions/25045709/… –  Radek S Aug 4 at 11:18
    
Robot doesn't have limitations in how it can be extended, as you imply in your answer. Robot is based on python, and you can write keywords in python or just about any other language, so it is even more extendable than a pure python solution. And while I don't expect a PO to write tests, the requirement was for a PO to be able to read tests, and robot is very strong in that area. Finally, robot uses selenium webdriver for web UI testing, so it has all the advantages of a pure python solution. –  Bryan Oakley Aug 9 at 12:03

I believe the difficulty you will have in making your choice is in how you define "simple" in a programming language. What you find simple may not be true for everyone. I always hear about how Python is simple and easy to read, yet I find it often more difficult to deal with than Ruby; and I do maintain a Test Framework in python so this is not coming from inexperience. Comparison for tools is difficult, especially when you have a list of "wants" as you have, I'd ask back to you first how your list got you to where you choices were Fitnesse and Robot. Other than they both have tabular displays, they are quite different.

For #1 - you can deal with any back-end testing with some slight programming, if you have taken the time to implement it. Both can do UI, Robot would need some additional Python to do the DB checks, but then so would Fitnesse. Even for UI Testing I needed to implement some additional checks using Selenium and Fitnesse. Technically it's feasible, whether it will meet your second option, and be simple, is debatable.

With #2, I would just reiterate what I said previously, how you define simple will matter here. Some non-programmers can read code, and some testers as well, and some have more difficulty. Product Owners probably won't be able to read the code, but if you get them a report that will allow you to show acceptance criteria coverage.

Jenkin's integrates with many tools, so you should be covered for #3.

Versioning should be supportable if you implement the cases appropriately with flags, tags or some other option. Robot handles Tags well, Fitnesse you can define what cases get run, and if needed both support documentation so you can keep track of changes within the cases as well as within any source control you may want to use.

Technically any integration is possible, if one does not exist there is usually a way a manage integration between tools - its a time, effort and resource problem.

You should check with your Team to see what they have preference, and experience, with. Check your developers, if you intend them to use these tools, to see what they also have experience with - or know what languages so you can leverage tools, and coding they have in place for your work. I've used Fitnesse and Robot in different jobs, they both fit the tasks they were chosen for, if they will meet your needs will depend on how they compliment the skills you have on your team.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.