I have and automated test that test an algorithm and the output of this test is the percentual precision of the algorithm. Currently our team is trying to make this algorithm more precise, as well as doing some refactorings. I need to find a way to do some kind of assert given the precision. My initial idea is just set a threshold (say, 80%) and create a assert like Assert.IsGreater(precision, 80). Although it seems OK, I wish I could have a better measurement tool than this. For example, I wish I could create levels (or grades), and set my precision according to these levels (100 to 90 is "A", 89 to 80 is "B", and so on). This would help me to control the quality of the code that is being built. Have you used any tools (or created one) to do such measurements? Did any of you passed for a situation like mine?
-
1It is difficult to answer your question without understanding the form of the algorithm's output and what the output should be compared to. What form does the algorithm's output take: e.g. a single number, a vector, a matrix?– user246Sep 7, 2011 at 13:11
-
I'll do my best to say something about the algorithm without giving much detail about it: the algorithm receives a matrix and the output is another matrix with some operations applied. There are some things that must be taken into consideration, as precision and performance. I can just measure the precision of the result knowing the exact result, spotting the diferences and using some heuritics to calculate the precision.– Filipe RodriguesSep 8, 2011 at 12:57
3 Answers
First I will summarize your question. You are testing an algorithm that takes a matrix as input and produces another matrix as output. Your test needs to measure the precision of the output. You do not want the result of your test to be simply past/fail; rather, you want your test to express a more graduated result.
I perceive two issues: how do you compute the precision in a way that measures quality, and how do you report the precision effectively?
Early in my career, I worked for a company that wrote software for the petroleum industry. The software could convert between digitized contour maps and matrices representing underground surfaces. It also had algorithms for filtering and combining matrices in different ways. There were different ways to measure the quality of the output of these algorithms, and to some degree, the quality was subjective. To put it another way, there were ways to quantify things about the matrices -- to assign a numeric score to the matrix, so to speak -- but the degree to which the score correlated with quality was always subjective. Moreover, I do not believe the scoring systems we used at that company were necessarily applicable to matrices in other industries, e.g. to digital images.
I say all this to illustrate that it is difficult to suggest an appropriate way to compute precision, as you call it, without knowing more about the data that your matrix represents and the goal of the algorithm. Nonetheless, I think it is a good idea to reduce your precision measurement to a single, concise number (or perhaps a vector) rather than a "precision matrix".
Regarding effective ways to present precision, I think the presentation method will depend on the context. If you are presenting a single precision by itself, the precision number may be good enough. If you are presenting a large number of test results, where you are comparing one precision number to another, it may be better to grade the precision with letters of the alphabet (as you mentioned), or with colors or shapes. I am sure a Google search would reveal websites devoted to effective ways of presenting statistics.
This question may also be relevant to you: How should I test the output of image renderers thoroughly, efficiently and in a branchable way.
your question is too vague, can you reveal more details about this algorithm ? I have been there, but the algorithm already had a way to test its precision (PSQM)
-
I can't reveal more details about the algorithm, but you have just reached the second problem of my testing: I can just test the precision with heuristics instead of certified quality measurements. Sep 8, 2011 at 12:49
You could have your test code log the precision measurements into an external storage (CSV, XML, RDBMS, you name it) and generate a report from that by an external tool. So if a refactoring brings the precision down, your external tool could do fancy stuff like sending everybody an XMPP message, calling the boss or shutting off the coffee machine until the former level has been restored.
Another idea is to take the threshold of the Assert from the last value written to the external storage and update that if the code passes the test. This would give you some sort of sliding threshold.
