Is it worth having one set of tests for both functional and performance testing?

Question

I have been writing RESTful service tests using Jersey client. Though the tests are functional, it occurerd to me that I can execute same tests from JMeter and for performance (With some tweak of course like logging to capture desired time response only b/w certain operations).

If not this approach then I would be writing RESTful service tests in JMeter using http sampler, with duplicating almost all the operations which I have covered in my functional tests.

The advantage I see here is the ability to have one set of test for both functional and performance testing.

I am curious to know if any other group tried this and has some recommendation on do's and don'ts

Answer 1

Performance tests and stress testing are different. I often re-use web service tests to add to my stress tests to add additional load to the system under test, but usually have a different set of performance tests to measure the performance of specific actions.

Usually these are run in tandem because I want to know how the API performs under no load, medium load, high load, varying load, etc. As for whether you should use jmeter or jersey client, which would be cheaper, adding a timer to the jersey tests or creating new jmeter tests? Also keep in mind that any additional overhead can affect your ability to create load, even something as simple as writing a debug statement to file can considerably slow down execution of your tests if you are trying to create a large amount of load. So, if the jersey client tests have a minimal footprint it shouldn't matter, but you might want to do some tests to compare.

Answer 2

I have tried it and at times gotten some use out of it, the problem I have found when doing this is that often the scenarios I want to use for Functional Tests don't exploit enough of the system to really give me any meaningful performance metrics. My functional tool also doesn't allow this as it was chosen for its purpose, while my load test tool was chosen for its purpose; if yours can handle both then great!

When I write Functional Tests I am just trying to exercise one aspect of the system repeatedly to make sure it passes the requirements, which is fine, and sometimes I get creative and work in some additional tests to get other areas tested as additional checks. That sometimes works. Although when I do performance profiling I am often looking to exercise other aspects of my product as there are areas that we know, or suspect, have issues with either load, load under performance or just performance and to exercise those adequately I need more than just a functional test.

It's not to say you can't resuse scenarios, or functional tests and I don't know the Jersey tool, but I think what you have to consider is whether you can combine the tests in a meaningful way to correctly exercise your system so you get useful measures back to determine your performance. If you can exercise your tests in parallel, which my functional tool can't, then you could probably work it in your tool as well to be able to get some additional performance checks by doing compatability testing and running tests that exercise other areas of your product in parallel with the area you are concerned about.