Do you still need a Test Plan when adopting Continuous Delivery?

Question

As a test manager, I need to review a Test Plan for the upcoming release. I also need to review the test results. The tests are both automated and manual.

As an organization, we would like to move to continuous integration/continuous delivery, where code changes are fully tested automatically e.g. via Jenkins. In the future, we may move to CD to automatically ship those changes

Do you still need a Test Plan when using CI/CD? How do you ensure that the product has been tested properly, including functional and non-functional testing? What oversight can QA provide when using CI/CD?

Answer 1

I don't see CI/CD as something different than non CI/CD process, tests are simply being run by a non human entity.

Do you need a test plan (or plan your tests) ? of course you do, you need to think what goes into automation and whether something should be done manually or tested in production (A/B tested, telemetry), analyze risks related to testing this way and not another, and evaluate the level of testing in each stage.

You can monitor the quality of the results the same way as before- use reports, dashboards and analyze logs, data and user bug reports

Answer 2

Do you still need a Test Plan when using CI/CD?

YES you do. Because a test plan will tell you at least the thing that you did not specify in your question: how often do you plan to run the tests? Which tests will run with which periodicity?

Maybe there is a common practice to do it every night. Is it feasible for your project? Which alternatives do you have? Why is one method better / worse than the other?

Which tests will be executed "continuously"? Top level / features-related? Architectural? Unit tests?

Can you test everything automatically? Does your system depend on other systems? How are the changes of the other systems managed and tested?

Even if "continuous" sounds so great, it is not at all continuous. At best, it is periodic. That is why you actually need to plan ahead.

Another question: what will happen when some test-case(s) fail? How will you handle the logging, reporting, monitoring, fixing?

---

Continuous vs. Periodic

A river flowing is continuous. Even if there is ice on the top of it. Testing cannot be continuous.

The code must achieve certain "maturity" before it can be tested. The minimum pre-requisite is that it is compilable.

The minimum pre-requisite for delivery is that certain test-cases must pass. If they fail, the product is not safe for use.

A test-suite takes time. You do not execute everything at any point in time. It starts, and then it finishes, and then a new SW version is taken from the repository, and testing starts again and ends again...

So such a system can be called "continuous" only if a lot of the practical details of reality are ignored - as it is usually done for marketing and advertising purposes.

Answer 3

No, you need a Test Strategy. A plan suggests it has a end, while your product SDLC is infinite. The test strategy should be a guideline for teams what types of testing they should execute and what value it brings, not how and when.

Some types of testing might need extra attention, as they do not fit in an automated build pipeline, some examples:

• Penetration testing
• Risk Analyses (for example the GDRP Data Protection Impact Assesment)
• Quality community
• etc..

Answer 4

As an alternative to "traditional" Test Plans (what we will do - checklist-based), you can focus on establishing a Test Strategy (how we want to do - guidelines-based).

A well-known model is the Heuristic Test Strategy Model:

The HTSM divides the testing creation thinking in different axes of analysis, which, when united, enables tester to create a holistic testing strategy. These axes are the following:

• Project Environment includes resources, constraints, and other elements in the project that may enable or hobble our testing. Sometimes a tester must challenge constraints, and sometimes accept them.

• Product Elements are things that you intend to test. Software is complex and invisible.Take care to cover all of it that matters, not just the parts that are easy to see.

• Quality Criteria are the rules, values, and sources that allow you as a tester to determine if the product has problems. Quality criteria are multidimensional and often hidden or self-contradictory.

• Test Techniques are heuristics for creating tests. All techniques involve some sort of analysis of the project environment, product elements, and quality criteria.

• Perceived Quality is the result of testing. You can never know the “actual” quality of a software product, but through the application of a variety of tests, you can make an informed assessment of it.

As a servant leader, instead of periodically reviewing (work done) and constraining how the team will work, you can be an outside eye, focused on making question that will make the team reflect if their action are aligned with the HTSM.

This way the team itself will become a self-organizing entity, capable of understanding its context and performing the work that needs to be done (which frees you to do something else).