Say I want to compare the performance of Java's StringBuilder.toString() with new String(char) for strings from one to 3000 characters in length. I do know beforehand how long the resulting string will be. How many times should I try each length, and how can I collect the results into usable information? I'm mostly interested execution time, but if there are any memory usage quirks I'd like to know about them also.
You didn't say how much precision you need -- do you just want to know which is faster, or do you want to know how much faster?
Here are some factors that interfere with benchmarks like that:
In any case, I suggest figuring it out empirically. Write the benchmark with a sample size of N and write down the results. Repeat with larger multiples of N, e.g. 1000, 10000, 1000000, etc., until the results converge to as much precision as you need.