5

There's someone who oversees QA - let's call him Joe. During 2 years of development we haven't had real load tests. Now release is getting close and QA is responsible for preparing and executing load tests. We (dev team) have suggested, that the load tests should be prepared and executed ASAP, before UAT starts and re-executed at least once before release.

Our proposal has been refused by Joe. The reason was that code is still changing and every code change could possibly break test scripts and require them to be updated. Joe explained that load tests should happen after code freeze, preferably after everything is already approved by the business. It was also refused by head of QA for the same reasons.

I'm not really into testing, so - is this really the way to go? Is it common practice to load test just before the release? Or are they all insane?

Improve this question
6
  • Just curious, are test scripts really coupled so tight with the code? Do they call some internals of the app, or only one external interface?
    – dzieciou
    Sep 15, 2013 at 8:57
  • They don't exist yet. They are supposed to simulate users clicking through a webapp.
    – user5985
    Sep 15, 2013 at 8:59
  • 2
    Two years of development and no automated test scripts so far? How do you make sure you do not have regression (i.e., you have not broken) functionalities you had already completed?
    – dzieciou
    Sep 15, 2013 at 9:14
  • +One more question. Is there any part of the app you are more suspicious about performance problems and that is already completed?
    – dzieciou
    Sep 15, 2013 at 9:16
  • 3
    Code changes can affect automated tests, it is a valid concern, however if you create your automated tests in an intelligent way, making sure they are maintainable and properly abstracted then it is almost always worth it to invest in them as early as possible. Perhaps if your QA team is inexperienced or uncomfortable with automation, your team could help them get the ball rolling by putting a maintainable foundation in place and some education.
    – Sam Woods
    Sep 15, 2013 at 15:51

4 Answers 4

Reset to default
4

You have been developing this system for 2 years and have never performed any load testing at all?

Preparing for expected load is usually part of the design. At this point, you are 2 years in, and may not have any evidence that your design is capable of supporting the required load. You do have performance and load requirements, right?

With any moderately-complex system (such as one that takes 2+ years to develop), there are always layers/components that can be tested at virtually any point in the development cycle, and which may cause performance and/or load problems.

At the risk of being mean, I think you are all insane (or at least negligent), not just the tastefully-named QA lead. I agree with Joe that some load tests should happen after code freeze, preferably after everything is already approved by the business. I just violently disagree that load tests should happen only after freeze.

Does Joe have trained performance/load testing people siting around with nothing else to do, waiting for code freeze? Or is this perhaps a resourcing or money issue?

If you haven't already done so, try to get resources devoted to analyzing the performance/load risk right now, and assess where you are. Perhaps, due to past work on this platform, management is comfortable that there will be no problems. Perhaps not.

Improve this answer
2
  • (It is always a money issue!) Official load tests happened twice in the past, but I choose to ignore them as they were useless (generated load 100x lower than on prod in the scenarios which matter). Unofficially something has been tested by us (development), conclusions drawn and some problems solved. No, we don't have any real performance requirements. The only defined "performance" metric in in 90% dependant on the human factor. So... I guess everyone here is insane (including me...). And no - management is not comfortable with performance of applications here, they are rather annoyed.
    – user5985
    Sep 17, 2013 at 0:53
  • 1
    @mabn - you choose to ignore them? They didn't tell you anything of value? Hmm, perhaps instead of saving money waiting for one big-bang load test at the end, it would be better to strategize how to get value from lower-cost tests earlier in the development cycle.
    – Joe Strazzere
    Sep 17, 2013 at 14:44
6

If the system needs to scale to many concurrent users or work with lots of data, I would say load testing should be started as early as possible. This way possible problems in the application architecture can be found in a phase where it is still possible to fix them.

Couple examples:

  • Let's say that the database schema is such that some queries will be slow with a large number of rows. Computers are nowadays so quick that nearly any query is fast with small amount of data and problems start only with a larger datasets. You really don't want to change database structure just before release. Load testing can be started as soon as database structure is decided, no need to wait for GUI to be ready. If the structure changes, earlier performance results can be used to evaluate the impact of the change.

  • Similarly problems in GUI may occur only with load. Maybe some third party library seems fine, but actually sorts stuff with bubble sort. Maybe another is not thread-safe but crashes or returns wrong data with certain timings. Maybe separate parts accidentally use same mutexes and cause deadlocks.

  • Maybe the system is designed for 1000 concurrent users. How it behaves with 2000 users? Does it crash, became unusable slow or gracefully serve some users and ask others to try again later?

It's not really possible to find these problems without testing. In my experience fixing this kind problems take time. If they are tested only just before release, any problem can cause delay in release schedule. As with nearly everything, test as early as possible, even if it's just a part of the whole system.

Improve this answer
2
  • +1 for examples. Should load testing focus on UI+backend, or only on backend?
    – dzieciou
    Sep 15, 2013 at 10:29
  • 2
    I would say that whatever may have load in the real situation. Back-end is almost always good candidate. On the otherhand slowness of GUI may be really irritating even if components work in the end.
    – Edu
    Sep 15, 2013 at 10:38
4

A few thoughts to add to the excellent suggestions others have made:

  • Yes, full end-to-end load tests should ideally happen after code is stable. These will be used to find bottlenecks in the interfaces between different components of your code. This kind of load test is probably what your QA people are thinking of when you say "load test" - and I'd hazard a guess that they aren't that experienced with load tests in the first place. It's a distinct sub-area of testing that many otherwise experienced testers and automators can find challenging.
  • Before the end-to-end load tests, your organization would ideally have performed more isolated load tests on individual components of the application. For example, Website A communicates directly with Database B. Web service C picks up orders made on the website and sends them to Web service D for processing. Web service D processes the orders and updates both Database B and shipping database E. It's not unreasonable to assume that the communications interfaces for the web services will be sufficiently stable to load test - which will tell you before code freeze if there are significant design issues under load.
  • In any project large enough to require a 2 year development effort, there should be a degree of designing for load. This should include indexing of all searchable fields in the database or databases, multithreading requirements, concurrency and lock protections, and the like. Code inspection can be used to find potential danger points, and specific loading tests used to ensure those danger points will perform under load.

I'd suggest that if you haven't already done so, you (as a dev team representative) offer to build component-specific load tests for the elements of your application and make the results of these available to the testing team. You might also offer to help with the development of load testing prior to code freeze so that you don't find your team scrambling to re-engineer fragile sections (which if the design is solid should in theory not be an issue - but theory isn't reality and chances are issues have crept in.

One final suggestion: when you're discussing this with the testing team, remember to frame your suggestions and questions as "We're all in this together: we all want a good product going out the door". It's too easy for differences of opinion in different teams to become points of contention, and you don't want that. (Apologies if this seems insultingly obvious: I've seen it get lost in the stress of a major release far too many times.)

Improve this answer
0
1

I would ask a number of questions before deciding when and what to load test:

  • Where performance problems tend to occur more often in your project: at UI, backend, or integration of UI with backend?
  • What part of your system is more stable? Horizontally, backend services or UI? How stable is the API between frontend and backend? Are there any integration problems that will result in changing backend implementation? Those parts may not be the best candidates for load testing now.
  • What functionality is the most stable, across all layers (frontend and backed)? They could be good candidates for more final tests.

UI tends to be an object of more frequent changes than the backend services API, so I can understand why your is QA reluctant to automate testing of that part. Also, my feeling is (could some expert confirm or reject that, please?) that load testing focuses on backend testing rather than on frontend part. Still, there are design patterns, like PageObject, that make your UI test scripts more resilient to UI changes.

Also, given the fact that you already have automated tests for the backend services, you could reuse those tests for load testing. That might be also easier to arrange in your organization, given the fact, devs are willing to load testing now and are owners of automated scripts for the backend services.

Finally, if you haven't done this so far, you may start sharing your automated tests with QA. Having the results of those tests, she or he could review both the tests results, test cases quality and coverage, and become more confident about stability of the system. And then, possibly, limit manual tests to areas are need more attention.

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.