Which testing approaches are most relevant to the actual production release of software ?

Do these methods include approaches to iterate to improve them ?


It's important to have considered the following testing methods:

  • human testing for issues that automation will miss
  • automated testing for the issues that humans will miss
  • complete run of current test suite on a CI environment
  • smoke testing when the release is live to make sure it works
  • testing on the devices that you know users use but devs (may) not
  • whether server volume will be affected and if load testing has been done
  • end to end tests on a staging environment that closely mimics production
  • well tested procedures to easily and efficiently roll back the release if there are issues

It is also important to consider:

  • production release and monitoring by a small % of live traffic initially
  • clear communication of go / no-go and methods used to transmit the decision
  • bringing up general process issues that arise during agile retrospectives (iteration)

This is how we do it::

We use manual test suites and automation test suites to test production releases.

Manual test suites for release testing:

We maintain 2 test suites for Release testing:

One for production and the other for pre-production environment. We are highly relying on these suite for testing.

Process of creating test suites:

  1. While creating manual test cases for user stories/requirements , we set the priority of each and every test cases and priorities are: P1, P2, P3 and P4

  2. All the test cases are executed on local testing environment as part of user stories / requirements.

  3. Then application is deployed in pre-production environment. In this environment; P1 and P2 test cases are executed. In our case, the application that we are testing is a web application and we are using MTM (Microsoft Test Manager) which allows us to create multiple configurations. I mean if you choose to test the application on Firefox, IE and Chrome; then your test cases will increase by 3 folds in this case, one test set for each browser). We select browsers on the basis of what we have committed to the customer (referring to the Test Plan document).

  4. Finally, once application is deployed in production we test P1 test cases. Here, also we configure the test suite for different browsers.

So, in every release, we keep on adding P1 tests to our production test suite and both P1 and P2 test cases to our pre-prod test suite.

How automation helps in keeping the size of test suites almost constant?

Our automation team is generally behind the manual testing team. So, automation team is working of automating the test cases that have already been tested during manual testing.

  1. So, we periodically review our manual test suites and remove the test cases that have been covered by the automation testing team. That way we keep on adding test cases for the latest release and remove test cases for the old releases that have been automated.

  2. Apart from this, we also review the suites for duplicate test cases (since some functionality might already be covered in some old P1 test cases). We also remove some of the cases which have been there in the test suite for quite some time and have lost relevance over time.

So, in Pre-production, automation suite is executed first. If, everything looks good then manual testing team jumps in to test the application using the test suites that we have.

In production, only smoke testing is done using automation suite created for production. This suite does not contain any scripts which can make changes in the application because we don't want to make any changes in production data. So, it's a subset of pre-production suite.

This has worked very well for us. There are hardly any high severity issues reported in production after testing. But in case, if a high severity issue is reported, we do the RCA for such issues. And in case if this have been missed during testing, we create a test case and add it to our test suites.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.