How to share responsibility for pipeline tests across multiple teams

Question

We have three teams working on the same product. Each follows the same sprint cadence but has their own scrum meetings.

Recently the API testing has shifted to developers mostly, including maintenance.

However, pipeline results are currently still monitored and delegated by the QA automation engineer. When a test fails, the responsible developer is addressed to investigate.

The obvious next step would be that developers take full ownership. This means a daily check of any failing tests, investigating and fixing (ideally whoever broke it). However considering there are several teams (and only one pipeline), we don't want multiple teams to do the same investigation, that would be a time-waster.

Any suggestions for a good setup?

Answer 1

This is one of those situations where there is no "best" answer. The one that will be most useful in your situation is a solution that everyone involved is willing to work with.

That said, I have a few comments/observations that may help you to decide which way to go.

• No changes - As I understand it, you have three teams contributing to a single product and a single CI/CD pipeline. With a third party checking failing tests, none of the development teams is unduly burdened with the checks. The automation engineer who is doing the checking may be better equipped to determine whether a failure is a false-positive (e.g. a timeout due to the product taking longer than usual to initialize) or an actual issue that needs to be fixed.
• A different third party analyzes the test results - In the same general spirit of the QA automation engineer analyzing the results, someone else who has the knowledge to do the analysis is also a possibility: a QA lead, a development lead, someone in DevOps... Whether this is an option for your situation depends on your employer's organization and the willingness of someone other than the automation engineer to analyze the automation results.
• Someone from one of the dev teams - If they are willing, someone from one of the dev teams could analyze the results. I don't recommend that someone who isn't willing be assigned to this: someone who's unwilling tends to be less likely to do the extra digging that's needed to determine whether a test failure is an issue or a false positive.
• Spread the joy around - Avoid a permanent time sink to one of the teams by having the task shift through the development teams on an agreed-on schedule. Team A might be responsible for one week, then Team B, then Team C, then back to Team A using this method. The risk here is that you won't get consistent analysis, and keeping the schedule posted and up-to-date becomes its own administrative time sink. On the plus side, this method ensures that over time all the developers become somewhat familiar with all the test automation and the CI/CD pipeline.
• Everyone checks - Instead of one person analyzing everything, everyone has the responsibility of checking the CI/CD dashboard and analyzing any failures that relate to their code. This method will take longer at first, until the devs become more familiar with the test automation and interpreting results, but pushes responsibility towards the individual developers to check the test results once they've integrated changes. This method requires all the developers to take full responsibility for their code and any downstream effects it might have (and as such, could be considered the "ideal" for a quality-conscious team), and as such won't be as effective as it could be if you have developers who don't see checking test automation results as part of their responsibility.
• A blend of all of the above - Your best option might be to mix and match from some or all of the potential paths until you find something that works well for your team and doesn't place too much of a burden on any one person.

Answer 2

The solution that worked for a team I worked with in the past was to hook the tests up to the pipeline so that they ran on every pull request. That way the developer who broke them is notified immediately because their build is broken. All tests had to be green before they could merge their work.

The pipeline was setup to run all unit tests on every PR.

For the integration tests, we organized them into test suites and then used a keyword based map to determine which suite of tests to run on a given PR. For this project, it was fairly simple because of the way the directories were organized. We could assume that if a developer made a change in a certain directory, we needed to run the related test suite.

In addition to this we had a smoke test suite that covered some important areas that would run no matter what. Meaning, even if someone changed something in a directory that did not exist in our map, the PR would still run the smoke test suite at the very least.

Once we implemented this, it became much easier to determine who broke it and who was responsible for fixing it. The only time the devs would reach out to me for help is if there was a major issue they could not figure out on their own. In most cases, I think it was fairly easy to see what went wrong and how to fix it.

For the keyword setup, we initially tried to make it so that devs could manually add tags for the suites they wanted to run on a PR. What we noticed was that no one was actually using the tags. So, we automated it, and added a step to the pipeline yml so that the tags were automatically added if changes were detected in the directories we included in the map. This was all done using Github Actions. Not sure how or if it would work using other similar services.

Answer 3

I am in a similar situation. We assign ownership out round robin style, so each team owns the responsibility for a week. A daily test update is provided in a channel for all teams to see, making the transition from one team to another relatively easy.

Answer 4

I'm kind of working on something similar. Have a suite of tests running every day for multiple applications in the same suite in a single pipeline. Here's a couple of things that I've found worked well.

Pipeline Setup - If you can, separate the tests by application or ownership into different stages in the pipeline. This makes it much easier to separate who to send work to when a test fails. Stages can be run and organized by test assemblies and or categories.

Test Setup - If you can, append the full request and response to your logs. Include a link to the application logs if possible. The request (including headers) means that it can be reproduced outside of the test. Response, including status code and response payload helps understand what's happening. The link to the logs is nice, but, with all of the included information, they should be able to find it very quickly.

Notifications - I like to have a slack channel where notifications can be sent. Only send notifications when it fails. If it's 100%, don't notify. Do your part to prevent notification fatigue. Make sure that the stage(s) that failed are mentioned so only teams responsible for those applications will investigate. Ideally, when they look at that stage, they can see exactly what test(s) failed, exactly what was used, and maybe a link to the log.

Next thing I'm working on figuring out is automatically creating a bug ticket on the board for the team who owns the application tested in that stage.

I've found with a couple of steps, the developers spend less time investigating issues and continue to be able to be responsible for their own testing. I spend less time with maintenance, and less time investigating issues, giving me more time to work on other automation initiatives.