3

Test frameworks help in test automation. However, as any software, they can and they do contain bugs. Just to give examples:

What metrics one should use to choose the framework that it credible and reliable?

Some I can think of:

  • Test framework itself has high test/code coverage
  • Reported bugs are fixed quickly
  • Framework is used widely in the domain similar to the one I want to use it
  • It is easy to fix some bugs on my own (the code is clear, authors accepts contributions and bug fixes)

What else?

4

All useful software has bugs, and neither the length of the bug list nor the unit test code coverage will tell you whether a test framework meets your needs.

Where I work, groups use either JUnit or TestNG for Java unit testing. We chose them because they are widely used, and so most of the common bugs -- the ones we are mostly likely to encounter -- have already been fixed. Any common bugs that are not fixed are probably well-documented in release notes or in something like StackOverflow.

If I were selecting a unit-testing framework from scratch, I would consider the following:

  • Is it open-source? (binary metric)
  • Is the documentation adequate?
  • Are there lots of examples written by people other than the authors?
  • Is it easy to get answers to questions, preferably because someone else already asked them and answered them.
  • When people answer questions about the framework, do they tend to sound satisfied with it, or do they sound angry/frustrated?

The question specifically asked for metrics, and most of those points are not phrased quantitatively, but I think you could find ways to quantify most of them.

Edit

The Op specifically wants to know whether test/code coverage will tell them anything about test framework reliability. I would be concerned about a framework with no unit tests, but if I were comparing two frameworks that both had unit tests, I would be reluctant to choose one over the other based solely on which one had more test/code coverage.

  • Do you mean the test/code coverage will not tell me anything about test framework reliability? That was the question. – dzieciou Feb 15 '16 at 6:53
  • @dzieciou Sorry, I misunderstood. I thought the question was, "What metrics one should use to choose the framework that it credible and reliable?" – user246 Feb 15 '16 at 12:27
  • you're right, that was the question. But don't you agree that test/code coverage is one of those metrics? – dzieciou Feb 15 '16 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.