How do I compare the performance of two .toString() methods?

Question

Say I want to compare the performance of Java's StringBuilder.toString() with new String(char[]) for strings from one to 3000 characters in length. I do know beforehand how long the resulting string will be. How many times should I try each length, and how can I collect the results into usable information? I'm mostly interested execution time, but if there are any memory usage quirks I'd like to know about them also.

Answer 1

You didn't say how much precision you need -- do you just want to know which is faster, or do you want to know how much faster?

Here are some factors that interfere with benchmarks like that:

• Clock granularity. I assume you will use something (System.currentTimeMillis, System.nanotime(), or a stop watch or whatever) to measure the time interval. The more coarse the granularity, the more samples you need to take. If this were the only confounder in your measurements (it isn't), you could derive a sample size from the granularity and the amount of precision you need.
• JVM behavior. The Hotspot compiler will convert code paths between bytecodes and machine code depending on how frequently they're used. The optimizer has to observe things for a while before it converts code. If your benchmark ends too soon, you will miss out on the optimization.
• Other activity on the machine. There are other things running on the computer besides your benchmark.

In any case, I suggest figuring it out empirically. Write the benchmark with a sample size of N and write down the results. Repeat with larger multiples of N, e.g. 1000, 10000, 1000000, etc., until the results converge to as much precision as you need.