1

I analyzing a static analysis tool. I have performed test on some samples apps and some real life apps.

I have to write comparison metrics for both sets of apps. But I'm not clear about what are the 'comparison metrics', especially for a static analysis tool.

Can some one please tell me what 'comparison metrics' means for a tool in general. And what will be the metrics for static analysis tool, will it consist of recall and precision?

A simple example will be really helpful.

I Googled about it, but just found the metrics for codes lines etc, and I could not understand what it will be in my case.

Improve this question
1
  • FYI, I'm not from QA background but I have to write report for a static analysis tool
    – blackfyre
    Commented Nov 17, 2014 at 12:36

2 Answers 2

Reset to default
0

Think about what you are analysing, what the tools do and what has your boss asked you to look at.

What features of the tools will be useful in your workplace? And, which will be a hindrance? What do the tools cost to buy and use: initially, per user, per year? What will be the cost of using the tools: what training is needed, how will they be integrated into your current working practices? Can you introduce the tool in a small way on a small project with a small number of users to get experience of it or must you do a "big bang" installation across everything?

You have done some test executions of the tools. Did they find the errors you were expecting? How many false positives were reported? How many errors were not detected? Can you turn off certain error messages so that today you can concentrate on the important messages but slowly over months or years turn on stronger checks. Can the tool perform strong checking on new code but weaker checking on old established code?

Try to rephrase any of the above topics that are important to you in ways that give numbers or Yes-No answers. Add other topics you find important.

Improve this answer
2
  • Should such analysis always be in tables? Can you give me a simple example
    – blackfyre
    Commented Nov 17, 2014 at 13:52
  • 1
    @user1204089 The question title includes the words "comparison" and "metrics". These suggest wanting an easy to understand representation of the results, a representation where similar attributes of each tool can be compared. This does not require a table, but a table can be a good way of presenting the results. Examples of suitable tables can be found on many shopping websites that invite you to compare a few products.
    – AdrianHHH
    Commented Nov 17, 2014 at 15:56
1

The categories that I would look for are based around:

  • Finding defects -> How many problems do each tool find. For each of the tools you will have to do an analysis and then dive into some of the results

  • Performance -> How much time does each tool take

  • Language -> Single or multi-language support

  • Standards -> are there any standards that you need to support

  • Usability -> Which tool has the better UI (very subjective)

  • Integration -> Can you integrate the tool into your environment

You'll have to spend some time to come up with metrics for these categories if hard numbers are important to you.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.