0

Baseline:

We work in a Safe-Scrum environment with a large number of different teams, and here also correspondingly different test teams.

We have different test cases from different teams, which are combined in our team. These include manual test cases, but also automated test cases.

From these test cases, we should now define various smaller test runs for the individual teams.

But also a complete regression test run that covers all teams.

In summary:

  • Many different teams in the Safe-Scrum environment
  • Our team runs all tests together
  • Both automated test cases and manual ones

SUT Overview:

  • The test cases are available both in the API area and as functional GUI tests. They are also automated in different Jenkins pipelines (for different teams). So we have more than 30 different pipelines, and this topic is accordingly extensive.
  • We have more than 50 teams (no joke) working on different topics for a major customer.
  • We developed most of the APIs test as Cucumber based test in a Java/Maven framework with direct integration to Jenkins pipelines and integration to X-Ray.

The problems I see:

  • We have a multitude of test cases within X-Ray that could not be more different. So there is the danger that you have automated test cases on one side and manual test cases on the other side, which leads to a complex and opaque situation.

  • There is a risk that we might constantly get new and old test cases that contain manual tests and automated ones and that we constantly have to work on hardening and making adjustments in the team.

Question:

How would you deal with this scenario?

7
  • 1
    There is a risk that we might constantly get new and old test cases that contain manual tests and automated ones and that we constantly have to work on hardening and making adjustments in the team. Constantly hardening and adjusting is standard in my experience Jun 23 at 10:56
  • 1
    You talk a lot about your test documentation, not much about the SUT. Explaining its architecture is probably important for any testing strategy. Jun 23 at 10:57
  • 1
    define 'properly' Jun 23 at 10:57
  • 1
    Yes it is a challenge. One set of tests providing one source of truth that things work may help in that challenge Jun 23 at 18:31
  • 1
    You might also want to consider moving away from SAFE-scrum as the thinly disguised waterfall model it promotes isn't very agile. Jun 23 at 18:32

2 Answers 2

2

Run all the tests for all the teams

In my experience, From these test cases, we should now define various smaller test runs for the individual teams. But also a complete regression test run that covers all teams. does not work, i.e. is not sustainable

A large amount of time would be spent organizing and tagging tests to create the subgroups. Then finding that some that were not run by team A should have been run so that needs to be adjusted and actually applies to team B as well, etc. Before long that becomes the work and the value actually added by the tests diminishes.

Insteads of this I recommend you focus on better tests that run faster. If ALL your tests ran in 1 second you would probably run all of them all the time for all situations and all user changes.

I would therefore focus on:

  • aviod e2e ui tests for negative test cases if a unit test can be used
  • mocking and stubbing expensive network or files system use
  • creating a test pyramid of unit, integrated and e2e tests
  • parallelize tests on multiple machines / threads
  • reliable and managed test data
  • coding to avoiding flakies
  • short bullet points ;)
4
  • The approach of smaller regression test suites and prioritizing testing, was also my thought. But that also means that you have to divide these suites according to manual and automated tests?
    – Mornon
    Jun 23 at 11:24
  • 1
    Manual and automated suites are not a 1:1 swap out, although they are often seen that way. There are some things that automation excels at, for example, filling out forms. Somethings are better suited to manual testing - is the color scheme harmonious, does the english used makes sense in the business domain we are in, etc. Jun 23 at 16:44
  • 1
    There is a similar question for unit vs. integrated tests - what if they overlap, how will we know which covers it, uses it, etc. The answer is similar, unit tests serve a very different purpose than e2e tests and coverage (code in one place, functionality in the other) should absolutely overlap. This is the same for manual vs automated e2e, they are not substitutes for each other. Also manual cases can be extremely complicated to set up for the data requirements and may be temporary so there is no (or negative) value in automating them. Jun 23 at 16:44
  • 1
    Once you have good automation in place you may consider traditional manual testing (all functionality possible) to be obsolete (the automation does it now) and should switch to exploratory testing with a Charter Jun 23 at 16:46
1

It depends

Assuming you have the support of your test teams and your management, I'd say that your best option would be to start incrementally improving your automation across all teams, while looking to define a minimum set of automated tests that must be run for all builds.

If the UI uses the API - I'd focus much more on the API tests to cover end-to-end functionality, since these will be faster than UI tests. In this scenario I'd keep UI tests to form entry/submission and checking that the data entered gets submitted with maybe a handful of critical-function end-to-end tests.

If you have access to the unit tests - I'd start with these, expanding unit tests so you have as much path/code coverage as possible. I'd also use the unit test framework to run lower-level integration tests to cover business logic so that the fastest possible tests were doing as much work as possible.

If the UI does not use the API - This leaves you with a lot fewer options. In this situation, I'd recommend carefully chosen core functionality be automated and supplemented with as much unit tests and integration tests as you can manage.

Where possible, eliminate manually performed, scripted regression - Try to automate scripted regression testing as much as possible. There are some things that should not be automated, such as hardware interactions, printouts, checking the appearance of the UI, and so forth. The rest should be automated as much as possible.

Schedule the automation - If you can, make your automated tests part of the build process. Your API tests, unit tests, and integration tests should be executed after each build. UI tests you might need to use a smoke test with each build and schedule daily runs of the full UI test suite to ensure your test runs don't become a bottleneck.

Work with developers - For unit and low-level integration tests, work with the developers. This may or may not be possible, but if you can it can be the best way to ensure you have good coverage that works well.

Clean up the manual tests - Again, where possible, any heavily scripted manual testing should be automated. Experienced manual testers are much better suited to be performing manual testing.

Make sure you have a searchable test repository - All your defined tests, whether manual or automated, should be visible to all testers and editable by testers who work with that part of the system. If you use a naming convention and start by having all tests renamed to match the naming convention, you will gradually find duplicated tests and be able to eliminate the redundancies. The manual testers should be able to see which tests are automated so they can focus their testing more effectively. Note that any naming convention will work, as long as it's consistent.

These are a few suggestions: no matter what you choose to do, starting and making small improvements constantly will gradually improve your situation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.