1

Within the Continious Integration stages, within this staging development, it is clear to everyone what is to be done where. Unitest in development, sonar if required, or RESTAPI testing via SoapUI.

  • development
  • various forms of automated tests
  • user acceptance testing (UAT)
  • staging
  • production.

Smoke tests and user acceptance testing (UAT) can be performed in a staging environment. Smoke tests check for essential service functionalities, and UAT tests are performed from the perspective of an end user. So far so good, but when should one not define functional test procedures within which test level?

Let's take load and performance test, which I can use after the developer stage in staging to simulate certain load scenarios in advance. At the same time, I can perform the same procedure weekly in production as a weekly performance check with information to stakeholders. Penetration test, on the one hand I can already identify security issues in the developer stage with SoapUI, and OWASP on this level. But also in the staging environment or of course what would probably make the most sense in production. All in all, a question of definition when I should use the non-functional means.

  • A question I would like to ask you here. Who made the best experiences with this kind of testing?
  • When do you use your non-functional tests?
  • How did you set up your CI / Staging environment?
  • Which single steps do you use when?

With best thanks Mornon!

  • 2
    Can you trim this down to one or two well-defined questions? As it stands it's basically asking to lay out a whole testing pipeline. – l0b0 Jul 24 at 6:24
  • 1
    I guess you're right, maybe the overall picture of the question is a little too powerful? Although they are self-contained questions, on the core point. I think about how I can change it into single questions. – Mornon Jul 24 at 6:27
2

The goal of testing is to prove that certain kinds of problems are absent from the product. But what if a test fails and uncovers that there is a problem? You want to know that as soon as possible, so that the problem can be rectified without costing the company too much money in terms of downtime, lost revenue due to late delivery, etc.

On the other hand, the different deployment environments also exist with a reason. The development environment may change too frequently for some tests, or it may have a too limited data set to work with. And when testing in the production environment, problems may be found too late and you certainly don't want to run tests there that might corrupt or destroy valuable data.

You have to find a balance there, and that balance does not come in the form of a blanket statement that all non-functional tests belong to environment X. Rather, for each (set of) test(s) you should determine in which environment it makes the most sense to execute them, given the tooling (or lack thereof) that you have to perform the tests and with the goal of finding problems as soon as possible without being hindered by the characteristics of the environment.

For example, tests that are performed by static code analysis or fast automatic tests can easily be done in the development environment. Running those tests doesn't take any effort from the testers beyond possibly setting up the tooling to run them automatically.

On the other hand, if the tests need to be done manually or they take longer to run, then the development environment may be too fast-paced for you and it would be better to perform those tests on the more stable staging environment.

Testing in production should be avoided as much as possible, because if you find problems there you are, by definition, too late. The main exception would be tests that depend on the exact configuration of the machines used in the production environment, like penetration tests.

  • Thank´s Bart, the fact that the test has to be mixed on the individual levels is of course absolutely correct. With this question I try to define when the non-functional test like Load/Performance Test or Penetration Test can be used sensibly. Of course, your statement is absolutely correct, and thank you very much for that! – Mornon Jul 24 at 7:48
  • @Mornon: Is the development environment similar enough to production to give representative results for a Load/Performance test? What would the customer experience be when you perform a Load/Performance test on the production environment? These kinds of questions should help you to determine by yourself which environment can be used most sensibly. – Bart van Ingen Schenau Jul 24 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.