Developer - QA - customer - production branching workflow

Question

How should one use branches to be able to test:

1. on local environment (during development),
2. on a branch for testing with real environment (QA-testing),
3. testing by customers on pre-production branch,
4. and finally merging to production?

We have the following branches: prod (customer production branch), dev(pre-production branch), staging(for QA), and local developers' branches.

For our current workflow, we have:

1. For development and local testing, done in developers' branches.
2. After code review, merge those local developers' branches to QA's branch.
3. After QA tested, merge changes to pre-production customer branch (customer will test)
4. Customer merges to production branch.

Problems appear on the second step. For example, we have some major tasks and some tasks that can wait. And developers already have done some major tasks and those tasks are already on step 2, but suddenly customers decide that some of the "can wait" tasks need to be done now and those changes should be merged right in pre-production branch so QA and customers could test it right away. Now customer liked all changes and he merges changes to production.

After all this we have:

1. prod and pre-production are the same.
2. staging branch is behind
3. developers always start new tasks from the production branch
4. a developer tested his task and wants to merge to staging (QA's branch)
5. also some other developers finish some "can wait" tasks and QA is already testing them, so they are in staging branch
6. But because branches are different he might have conflicts with other developers' work. But his work is a major task.
7. Now he needs to fix conflicts. That means to update some changes from staging in his branch and merge to staging for testing.
8. It results in a branch which started from production and has some chunks of code from staging (that have probably not been tested yet).

What we should do in such cases? What should a normal workflow look like?

Answer 1

The flow at every company I've worked for has gone like this:

local branches (branched off dev) -> dev -> test/qa -> staging -> prod

Keep in mind that during this entire process, automation is running after each deploy/merge.

1. Engineers branch off of dev, and merge that branch into dev once they have completed their initial testing.
2. If dev is considered stable, it is promoted to the QA/TEST environment.
3. Once we have feature freeze, the last promotion of DEV->QA/Test is done
4. Test is now "feature complete" for this release, and QA begins regression/feature testing, bug bash happens against test
5. If bugs are found in qa/test, fixes are merged into qa/test and backported to dev.
6. Once testing is done on qa/test environment, it is promoted to staging.
7. More testing, bug bashes, customer testing, etc on staging. Possibly with different scope than the qa/test testing. If bugs are found they are merged into test and promoted and backported.
8. Once staging has the green light we deploy to production.
9. After a smoke test on production feature freeze is over.

Ideally you want to separate the environments for what you are doing on them. the dev environment is where devs branch from and check in code. The test environment is where QA does initial testing after feature freeze. The Staging environment is not surprisingly where features are "staged" for demos, customer review, secondary rounds of testing, and finally approval.

Additionally, in the best case scenario, multi-sprint features should be split up into smaller pieces that can be incrementally pushed out with each release.

Devs should never be branching from production, and should be branching from the environment they will be initially checking the code into to reduce merge conflicts.