How to gather usage statistics for various open source testing frameworks

Question

My organization would like to choose between the TestNG and Robot Framework. I thought it better to approach this from the stand point of usage statistics, since both are open source. How do I go about doing this. I did not find any head to head comparison information when I looked it up and so I thought it better to go with usage stats. Could someone point me in the right direction?

By the way we're a product company providing solutions for the Supply chain Management domain. We have both J2EE Web applications (flagship) and mobile applications (iOS and Android)

Currently we're white-boxing with Groovy/testNG but there's confusion for black boxing. We're planning to use Selenium as the automation tool with either TestNG or Robot.

Answer 1

I think you want to pick the framework which has the most active development and the most documentation resources on the internet.

Of-course you first need to check which framework fits your requirements, I would pilot all for a short while (starting with the most active one. If you have multiple candidates.)

Which is more popular:

• Number of (recent) commits in the version control system
• Number of contributors in the version control system
• Number of tags on StackOverflow (Robot + RobotFramework:500 vs TestNG:1665)
• Number of Google results
• Active users on the IRC channel on irc.freenode.org

In the past we have picked a slightly "worse" framework, because it was actively developed compared with the single developer for the other framework.

Other things I would look into are:

• Is the framework in a programming language your team has experience with
• Does the framework have good test coverage

This will make it easier to extend it in the future. When the community does not fix a defect your are running into.