I'm looking for a tool/framework (preferably in Java if it involves coding) that allows me to load test a server. I need to be able to launch multiple EC2 instances, hit the server hard for a period of time, log the response times to something that can be graphed (i.e. CloudWatch?), and then shutdown the instances. Automation for the tests is also needed.
I haven't had a lot of experience with load testing - I've tried out using the Jmeter GUI for a very simple test to hit a URL, and I've been researching how Grinder works. Could either of these allow creation/shutdown of EC2 instances in their tests? Is there a better option out there to do this?