8

As a test manager I need to review a Test Plan for the upcoming release. I also need to review the test results. The tests are both automated and manual.

As an organisation we would like to move to continuous integration/continuous delivery, where code changes are fully tested automatically e.g. via Jenkins. In the future we may move to CD to automatically ship those changes

Do you still need a Test Plan when using CI/CD? How do you ensure that the product has been tested properly, including functional and non-functional testing? What oversight can QA provide when using CI/CD?

  • 1
    As the answers say, yes. However make sure that the focus stays on the value the tests add and is constantly reviewed. Test plans, although needed, can be the start of creating massive test suites (whether manual or automated) and it takes maturity to contually refactor and remove tests to keep the suite fast and lean. Just adding more tests every time there is new or changed functionality is a route to slowness. Consider KPIs that cap test running time to force choices. This will force more unit tests 'cos they are so fast. – Michael Durrant Oct 14 at 14:59
  • Details of approaching/utilising CI/CD for testing also should be a part of high level test plan. – Vishal Aggarwal Oct 15 at 11:15
7

I don't see CI/CD as something different then non CI/CD process, tests are simply being ran by a non human entity.

Do you need a test plan (or plan your tests) ? of course you do, you need to think what goes into automation and whether something should be done manually or tested in production (A/B tested, telemetry), analyze risks related to testing this way and not another, and evaluate the level of testing in each stage.

You can monitor the quality of the results the same way as before- use reports, dashboards and analyze logs, data and user bug reports

7

Do you still need a Test Plan when using CI/CD?

YES you do. Because a test plan will tell you at least the thing that you did not specify in your question: how often do you plan to run the tests? Which tests will run with which periodicity?

Maybe there is a common practice to do it every night. Is it feasible for your project? Which alternatives do you have? Why is one method better / worse than the other?

Which tests will be executed "continuously"? Top level / features-related? Architectural? Unit tests?

Can you test everything automatically? Does your system depend on other systems? How are the changes of the other systems managed and tested?

Even if "continuous" sounds so great, it is not at all continuous. At best, it is periodic. That is why you actually need to plan ahead.

Another question: what will happen when some test-case(s) fail? How will you handle the logging, reporting, monitoring, fixing?


Continuous vs. Periodic

A river flowing is continuous. Even if there is ice on the top of it. Testing cannot be continuous.

The code must achieve some certain "maturity" before it can be tested. The minimum pre-requisite is that it is compilable.

The minimum pre-requisite for delivery is that certain test-cases must pass. If they fail, the product is not safe for use.

A test-suite takes time. You do not execute everything at any point in time. It starts, and then it finishes, and then a new SW version is taken from the repository, and testing starts again, and ends again...

So such a system can be called "continuous" only if a lot of the practical details of reality are ignored - as it is usually done for marketing and advertising purposes.

  • 1
    "it is not at all continuous. At best, it is periodic" Do you mean one cannot implement delivery of every codebase snapshot, but only on specific periods of time? – João Farias Oct 15 at 7:06
  • 1
    @JoãoFarias At one level, changes to the code-base happen in discrete stages, so – in a somewhat mathematical sense – it can only be step-wise and not continuous. More generally, it will take a certain amount of time to run through a suite of tests, so a new "tested release" cannot be continuously available, only periodically available (with a minimum period based on the length of time the tests take to complete). – TripeHound Oct 15 at 8:34
  • @JoãoFarias: I added to the answer to clarify the "continuous" vs. "periodic". Sorry I did not do it from the beginning - what is in my mind is always obvious to me :) – virolino Oct 15 at 8:39
  • 1
    @TripeHound: exactly my point - we added the explanation at the same time. +1 – virolino Oct 15 at 8:39
  • 1
    Well, if we look at the atomic level, a river flows discretly. Everything will depend on the POV. In Continuous Delivery, a deployable unit is defined as a green commit. If you have a new deployment for each new green commit, immediately after its creation (coding+testing), then you are deploying continuously. If you accumulate green commits to be deployed together, you are not. – João Farias Oct 15 at 9:03
1

No, you need a Test Strategy. A plan suggests it has a end, while your product SDLC is infinite. The test strategy should be a guideline for teams what types of testing they should execute and what value it brings, not how and when.

Some types of testing might need extra attention, as they do not fit in an automated build pipeline, some examples:

  • Penetration testing
  • Risk Analyses (for example the GDRP Data Protection Impact Assesment)
  • Quality community
  • etc..
-1

As an alternative to "traditional" Test Plans (what we will do - checklist-based), you can focus on establishing a Test Strategy (how we want to do - guidelines-based).

A well-known model is the Heuristic Test Strategy Model:

enter image description here

The HTSM divides the testing creation thinking in different axes of analysis, which, when united, enables tester to create a holistic testing strategy. These axes are the following:

  • Project Environment includes resources, constraints, and other elements in the project that may enable or hobble our testing. Sometimes a tester must challenge constraints, and sometimes accept them.

  • Product Elements are things that you intend to test. Software is complex and invisible.Take care to cover all of it that matters, not just the parts that are easy to see.

  • Quality Criteria are the rules, values, and sources that allow you as a tester to determine if the product has problems. Quality criteria are multidimensional and often hidden or self-contradictory.

  • Test Techniques are heuristics for creating tests. All techniques involve some sort of analysis of the project environment, product elements, and quality criteria.

  • Perceived Quality is the result of testing. You can never know the “actual” quality of a software product, but through the application of a variety of tests, you can make an informed assessment of it.

As a servant leader, instead of periodically reviewing (work done) and constraining how the team will work, you can be an outside eye, focused on making question that will make the team reflect if their action are aligned with the HTSM.

This way the team itself will become a self-organizing entity, capable of understanding its context and performing the work that needs to be done (which frees you to do something else).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.