Should load test validate functionality?

Question

I am developing a loadtest to a customer who is saying that if a certain response contains "bla:bla" I should execute a different path in load test. On client side it means that a certain button is disabled and user should not be able to push it. On the other hand, if I ignore the mark-up and send it anyway, the server is perfectly fine with it (means it returns HTTP.200). So the customer is demanding that I check for this condition, for me I would mean that I had to parse all the responses of this type and decide which path to take, and this seems absurd. My reasoning is that the server should tell me to get lost. So the point is that data is not being validated on server side (which is a noticable security issue), but I have been forced to write a 'hack' to deal with this. I am not generally against parsing responses, ie, due to browser implementation, it might be totally ok to return unauthorized with HTTP.200, but the question is - to what extent should I tolerate parsing the responses?

Answer 1

This is a very strong "yes". You might want to limit yourself a little in the overall validation, but checking response codes only is a strong fail in my opinion. Additionally just checking the result by checking a single phrase or word is not enough.

Update: This response answers the topic question: "Should load test validate functionality?" and does not deal with the details of the post which slightly differ from the subject.

Reasons and Examples

• Modern web implementations often incorrectly return application status pages with response code 200
• How do you ensure that you got the entire page back and not just the first 75% only?
• Imagine an ecommerce search that breaks under load and instead of saying I found 200 matches, it returns a page saying "no matches found, did you mean that...". The later is still a valid page but your load test will not discover the flaw.
• If you implement real page flows aka you do not have hardcoded urls that you fire against the server, instead you take the data from the page and you apply randomness to get a good wide sampling, you might find pages that are incorrect or inconsistent, hidden outside normal functional testing click paths.
• How can you tell apart a "login succeeded" from a "login failed" page? Both are valid scenarios and login failed might not return a code different from 200. When designed to return "get lost" aka 401, it might break under load and incorrectly return 200 for a page with "login failed".
• Always assume the worst case, always assume that the application will behave differently under load. Not only timing-wise but also behavioral wise. This is one reason for running load and performance tests.

A Tale from my Past

In my previous company, a colleague was testing something for a new web component. He said, that the new implementation is better, because it got faster when he applied more load.

Ugh? Well, he did not check the content of the returned page. Under higher load, he got error pages back and incorrectly assumed great performance instead of a broken application.

How We Handle Validation

Because I am with a web testing company (Xceptance), specialized in load and performance testing, here is what we generally do for all our load and performance testing:

• Validate the response code
• Validate the basic structure of the page, check that the main components are there, such as header, footer, search box, mini cart, user link...
• Validate the completeness of the page aka the closing html should be there or the JSON or XML should be complete (hopefully also valid)
• Validate the most important points of the current activity, e.g:
  • you search for an existing product, you have to find it
  • you expect a non-result of a search, check that
  • you add something to the cart, validate the quantity change
  • you checkout, validate the order number format (under perfect conditions even the uniqueness or at least the change in between checkouts), validate that all price information is a valid format (not necessarily calculated correctly, that would be required under certain test conditions of course), check that the user name used (should be random for instance) is used during that visit aka you expect a name, so it should be that name (billing, shipping, order, account)
  • you log on, verify from now on, that you always see the name of the user on the page (the famous Welcome Mr. Foo or something like that)
  • verify that you do not see a Welcome Mr. Foo before login, that you do not have a cart when you arrive, and so on

We have so often found problems load related or often also functionality related that every single validation step really pays off again and again.

Only when applications return bad stuff and the implementation crew does not want to change that and cannot be convinced or forced to clean up the mess, we apply exceptions and reduce the validation.

These are just examples of course.

Verdict

You should extensively validate during load and performance testing. Always assume that the application will break under load and that is mostly not reflected in a status code 500. 404 and 500 are for the beginner, the real professionals find problems beyond that.

We usually find functional problems during load and performance testing. Often not during the load execution itself, but during low load dry-runs. Not to forget the classic erroneous load behavior, such as inconsistent response times, spikes, and everything else imaginable.

Or in other words: You do load and performance testing to find the unusual and unexpected. This is often not reflected in increased response times and error codes.

Of course there are tests, when you really hunt for something special, that you can easily skip most of the validations, but never strip them completely.

Answer 2

I think that the answer is "Yes", load test should have checks for response validity. I.e. if you send bad username/password pair it is expected to get 401 or 403 HTTP response and JMeter shouldn't consider it as sampler failure. If response code is 200 but the response body doesn't contain "bla-bla" although shouldn't it also should indicate test failure.

If basing on response test should go this or that scenario it also needs to be implemented as load test needs to be as much realistic as possible, it isn't just hammering single page, it is close simulation of real user's behavior.

So I encourage you to use IF Controller to implement conditional test flow and JMeter Assertions to define pass or fail criteria.

Answer 3

The simplest answer is if your customer is telling you to do this, you do it.

If you don't provide what your customer wants, you run the risk of them telling you after you've developed the test that the results aren't valid because you've ignored their instructions.

I've seen this happen: a very simple load test had to incorporate a lot of business logic before the company would accept that the results were valid, because they didn't consider a result that didn't accurately simulate customer experience a valid result and believed the results of the load test were corrupted by invalid data (in this case it was "they're not all going to buy the same product" and "some of them will log in, some won't", and "they'll be in different departments" and so forth).

Give your customers what they ask for. The person who's paying for the service is always "right" in that they get what they pay for. If you can't convince the customer of the merits of your preferred approach, you, as the contractor/employee/person being paid need to do what they ask whether you think it's valid or not (although if you think there's a security issue, you would do well to ask them if they've arranged for security testing and strongly recommend that they do so if they haven't. That's simply due diligence).

Answer 4

I would say that some validation is almost always needed. For example I have tested website where if the login failed then all subsequent requests get a 200-OK response, but every response body requests a username and password. Adding a simple verification that the response to the login contains "login successful" or something equivalent detects the login failure. Similarly, after sending a request corresponding to clicking a button it may be worth validating that the response contains text that shows the expected response page was shown.

So I recommend checking for basic functionality, namely that the correct page is shown or that the correct static text is shown. But I would not expect to check that the values shown on the pages are correct.

Answer 5

The load test should reflect the kind of traffic you expect to see in production. If that means a lack of button presses when a page is in a certain state, that's what you should do.

The fact that the server allows workflows that the client prevents is good to know, but apparently your customer believes most traffic will come from users who use the client as intended.