1

There will be a lot of releases coming out during the lifecycle of a package release. How to ensure that the latest change does not affect the previous components?

I would say once a release candidate is ready, a regression test is to be performed on the Release Candidate. However, if a bug were to be found, is it best practice to retest everything again once a new Release Candidate is released?

Wouldn't that be time consuming?

  • What do you mean by previous components? – dzieciou Jan 8 '18 at 8:25
  • I assume your first "regression test"(smoke suite) once release candidate is ready and later full blown regression suite in case bug found, are all being run through automation, is that assumption is correct? – Vishal Aggarwal Jan 8 '18 at 8:45
  • If yes, then what is the execution time for both of these automated suites? – Vishal Aggarwal Jan 8 '18 at 8:48
2

Exactly. Because there is no way to ensure that change in one component does not affect any other components, your only option is automated regression testing.

1

The answer is Continuous Integration. Good practice is having a continuous integration server watching the version control system so when commit is made it will trigger:

This way you will have confidence regarding your application robustness and reliability even given new features introduction and bug fixes. If you don't have continuous integration in place I would recommend setting it asap, there is a number of free and open source continuous integration servers you can consider, de-facto industry standard is Jenkins.

  • 1
    Automated tests (whether unit or load, or UI/e2e tests) do not have be run via CI. First step is to run all test on demand. CI is just gravy. – Peter M. Jan 9 '18 at 22:22
  • A lot of the jetbrains IDEs can be configured to automatically run tests when your code changes, this is a great way to start to do CI – Amias Feb 6 '18 at 10:27
1

Regression testing will certainly help here but it should be applied with a focus on the product risks. If you take some time to chat or pair with the developers to understand the scope of the changes it should be possible to identify the areas with the biggest risk of regressions.

When creating a regression suite choose the most reliable tests you have and skip any that might give questionable results (or clearly mark them as such), this keeps confidence in the results high and ensures developers take them seriously.

If you have the time and specs you can create at least some of tests ahead of development so they have a target (TDD) , this gives the benefits of CI right at their desk, however this isn't always possible with complex projects or less specified development styles. Also be very sure your test is accurate !

If you cant do TDD then at least try to deliver tests focused on the areas of riskiest areas change.

The earlier in development you can test meaningfully the cheaper it is to fix bugs you find.

. Continuous Integration is good way to keep code tested , it takes time and you need to write tests in a slightly more stable way than standalone ones. Generally you need s team of testers and possibly some DevOps to get the most out of CI.

So in summary

Cleanup, focus , test early

1

I quite agree with Amias and his proposed approaches to regression testing. He writes about risk-based approach and test suite maintenance to keep the regression test suite up-to-date. I can also add three more approaches to regression testing that make it less time consuming and more efficient. They are:

  1. Two-level approach breaking regression testing into 2 cycles:
    • iterative regression that centers on changes made during the iteration and adjacent areas
    • major regression which features the complete regression test suite. This type of regression testing is to be run before major releases
  2. Collaborative approach. With this approach, the team creates a regression dashboard, where team members shortly describe their work. Every team member can make changes on the board based on their knowledge about the project. When it's time to run regression, the team consult the board and assess the testing scope and the time needed. I guess this approach may work well for you, as your team is not too big.
  3. Regression testing automation. This is one of the most reasonable moves that reduces time and effort spent on this tedious type of testing. However, automation itself is a complex effort, so it requires an effective strategy to work well.
0

There is no silver bullet answer to this, you should use common sense.

Usually you would want to repeat some basic tests to cover basic functionality, add some tests around the fix and probable affected areas and if all goes well use the rest of the time for more tests to gain coverage.

You will also want to have some automation if test run time vs. release frequency is small enough, but be careful not to waste all your time developing automation and neglecting coverage or stability of the tests.

If your product is a web app with good monitoring and quick release time, so you can fix problems fast, I would feel comfortable doing superficial testing, on the other hand if you are in the aviation or medical section than your answer would be a big YES- test everything from scratch again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.