0

Baseline:

We work in a Safe-Scrum environment with a large number of different teams, and here also correspondingly different test teams.

We have different test cases from different teams, which are combined in our team. These include manual test cases, but also automated test cases.

From these test cases, we should now define various smaller test runs for the individual teams.

But also a complete regression test run that covers all teams.

In summary:

  • Many different teams in the Safe-Scrum environment
  • Our team runs all tests together
  • Both automated test cases and manual ones

SUT Overview:

  • The test cases are available both in the API area and as functional GUI tests. They are also automated in different Jenkins pipelines (for different teams). So we have more than 30 different pipelines, and this topic is accordingly extensive.
  • We have more than 50 teams (no joke) working on different topics for a major customer.
  • We developed most of the APIs test as Cucumber based test in a Java/Maven framework with direct integration to Jenkins pipelines and integration to X-Ray.

The problems I see:

  • We have a multitude of test cases within X-Ray that could not be more different. So there is the danger that you have automated test cases on one side and manual test cases on the other side, which leads to a complex and opaque situation.

  • There is a risk that we might constantly get new and old test cases that contain manual tests and automated ones and that we constantly have to work on hardening and making adjustments in the team.

Question:

How would you deal with this scenario?

Improve this question
7
  • 1
    There is a risk that we might constantly get new and old test cases that contain manual tests and automated ones and that we constantly have to work on hardening and making adjustments in the team. Constantly hardening and adjusting is standard in my experience
    – Michael Durrant
    Jun 23, 2022 at 10:56
  • 1
    You talk a lot about your test documentation, not much about the SUT. Explaining its architecture is probably important for any testing strategy.
    – João Farias
    Jun 23, 2022 at 10:57
  • 1
    define 'properly'
    – Michael Durrant
    Jun 23, 2022 at 10:57
  • 1
    Yes it is a challenge. One set of tests providing one source of truth that things work may help in that challenge
    – Michael Durrant
    Jun 23, 2022 at 18:31
  • 1
    You might also want to consider moving away from SAFE-scrum as the thinly disguised waterfall model it promotes isn't very agile.
    – Michael Durrant
    Jun 23, 2022 at 18:32

3 Answers 3

Reset to default
3

Run all the tests for all the teams

In my experience, From these test cases, we should now define various smaller test runs for the individual teams. But also a complete regression test run that covers all teams. does not work, i.e. is not sustainable

A large amount of time would be spent organizing and tagging tests to create the subgroups. Then finding that some that were not run by team A should have been run so that needs to be adjusted and actually applies to team B as well, etc. Before long that becomes the work and the value actually added by the tests diminishes.

Insteads of this I recommend you focus on better tests that run faster. If ALL your tests ran in 1 second you would probably run all of them all the time for all situations and all user changes.

I would therefore focus on:

  • aviod e2e ui tests for negative test cases if a unit test can be used
  • mocking and stubbing expensive network or files system use
  • creating a test pyramid of unit, integrated and e2e tests
  • parallelize tests on multiple machines / threads
  • reliable and managed test data
  • coding to avoiding flakies
  • short bullet points ;)
Improve this answer
4
  • The approach of smaller regression test suites and prioritizing testing, was also my thought. But that also means that you have to divide these suites according to manual and automated tests?
    – Mornon
    Jun 23, 2022 at 11:24
  • 1
    Manual and automated suites are not a 1:1 swap out, although they are often seen that way. There are some things that automation excels at, for example, filling out forms. Somethings are better suited to manual testing - is the color scheme harmonious, does the english used makes sense in the business domain we are in, etc.
    – Michael Durrant
    Jun 23, 2022 at 16:44
  • 1
    There is a similar question for unit vs. integrated tests - what if they overlap, how will we know which covers it, uses it, etc. The answer is similar, unit tests serve a very different purpose than e2e tests and coverage (code in one place, functionality in the other) should absolutely overlap. This is the same for manual vs automated e2e, they are not substitutes for each other. Also manual cases can be extremely complicated to set up for the data requirements and may be temporary so there is no (or negative) value in automating them.
    – Michael Durrant
    Jun 23, 2022 at 16:44
  • 1
    Once you have good automation in place you may consider traditional manual testing (all functionality possible) to be obsolete (the automation does it now) and should switch to exploratory testing with a Charter
    – Michael Durrant
    Jun 23, 2022 at 16:46
3

It depends

Assuming you have the support of your test teams and your management, I'd say that your best option would be to start incrementally improving your automation across all teams, while looking to define a minimum set of automated tests that must be run for all builds.

If the UI uses the API - I'd focus much more on the API tests to cover end-to-end functionality, since these will be faster than UI tests. In this scenario I'd keep UI tests to form entry/submission and checking that the data entered gets submitted with maybe a handful of critical-function end-to-end tests.

If you have access to the unit tests - I'd start with these, expanding unit tests so you have as much path/code coverage as possible. I'd also use the unit test framework to run lower-level integration tests to cover business logic so that the fastest possible tests were doing as much work as possible.

If the UI does not use the API - This leaves you with a lot fewer options. In this situation, I'd recommend carefully chosen core functionality be automated and supplemented with as much unit tests and integration tests as you can manage.

Where possible, eliminate manually performed, scripted regression - Try to automate scripted regression testing as much as possible. There are some things that should not be automated, such as hardware interactions, printouts, checking the appearance of the UI, and so forth. The rest should be automated as much as possible.

Schedule the automation - If you can, make your automated tests part of the build process. Your API tests, unit tests, and integration tests should be executed after each build. UI tests you might need to use a smoke test with each build and schedule daily runs of the full UI test suite to ensure your test runs don't become a bottleneck.

Work with developers - For unit and low-level integration tests, work with the developers. This may or may not be possible, but if you can it can be the best way to ensure you have good coverage that works well.

Clean up the manual tests - Again, where possible, any heavily scripted manual testing should be automated. Experienced manual testers are much better suited to be performing manual testing.

Make sure you have a searchable test repository - All your defined tests, whether manual or automated, should be visible to all testers and editable by testers who work with that part of the system. If you use a naming convention and start by having all tests renamed to match the naming convention, you will gradually find duplicated tests and be able to eliminate the redundancies. The manual testers should be able to see which tests are automated so they can focus their testing more effectively. Note that any naming convention will work, as long as it's consistent.

These are a few suggestions: no matter what you choose to do, starting and making small improvements constantly will gradually improve your situation.

Improve this answer
1

To properly manage regression testing as an interface between different testing teams in a Safe-Scrum environment with many teams and a large number of test cases, I would suggest the following steps:

Define clear guidelines for test case creation and management: All teams should follow a consistent format for creating and documenting test cases, including clear definitions of manual and automated test cases.

Create a centralized test case repository: All test cases from different teams should be stored in a centralized repository, such as X-Ray, to ensure they are easily accessible and searchable.

Establish a test case review process: A dedicated team or individual should review all test cases before they are added to the centralized repository to ensure they are accurate, complete, and follow the guidelines.

Define test run templates: Create templates for different types of test runs, such as individual team test runs and complete regression test runs, to ensure consistency and efficiency.

Implement test case automation: Automate as many test cases as possible, using tools such as Jenkins, to reduce the workload of manual testing and improve the overall efficiency of the testing process.

Regularly review and update the test cases: To minimize the risk of constantly getting new and old test cases that contain manual tests and automated ones, it's important to review the test cases regularly and update them as necessary.

Use of a test management tool like X-Ray : it can help you to keep track of all test cases and test runs, and gives you an overview of the test results. Consider using a tool for test case management like TestRail, which can help you to organize, manage and track all of your test cases, test runs and test results.

By following these steps, you can ensure that your team can effectively manage regression testing as an interface between different testing teams in a complex and extensive environment.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.