Whenever we think about the performance aspect of the Application/ Database/ feature or whatever. Never go for single results set or single iteration. You need to collect enough data for the statistical analysis of the result for reaching to an answer that your subject under test is performing Good or Bad.
It is simple fundamental of Statistics that "More data you have more precise/accurate your result will be" i.e. you will be more near to the correct result (with limit --> 0). Hence you need to collect data for multiple samplers. You can again relate this with the results provided by JMeter i.e. 90% line (for example), if you have only 1 sample then your Average, Min., Max. and 90% will be same and when you have more data all values will be different and same will happen with your Std. Deviation too and you will be plot a good graph for analysis and reporting.
So, answer to your question
Is it sufficient to run the test once for each configuration to take a
measurement (memory used, elapsed time, CPU time etc). Or should more
samples be required
is YES, you should have more samples.
These more samples you can generate using the JDBC sampler of Apache JMeter, for this either you can run your script for Single user but for Multiple Rounds (if you don't want to put concurrent user load on your scenario) Or you can run multiple threads at once.
Collect data for 400-500 samplers and that too multiple times like:
- Run your script for single user (i.e. No of Threads = 1, but number of rounds = 500)
- Run this above mentioned scenario multiple times a day e.g. once in morning, then in afternoon, 3rd one in evening.
- For more precision, you can repeat steps 1 and 2, for multiple days (3-4 days) and then compare & average out the results.
You can try tuning the Database between your various rounds of execution to see if that tuning and optimizing is helping you in +ve direction (that's how performance testing and performance tuning goes hand-in-hand), but never go with single sample or single round results only always have mutiple.