System integration testing

Question

I had some questions around system/integration test. We have a development team and a full QA team, which does customer focused testing for our product. We want to introduce a middle test team which is more of an extension of the engineering orr, a system integration test team if you will.

Thinking about the rules for what the developers are expected to test, what this new team is expected to test, and the final QA customer team. The problem it is supposed to solve is the QA team gets code which is not really tested well, and falls apart right away, since developers are doing unit tests mostly, and QA is doing scale, customer focused scenarios. We are doing smoke to take care of the immediate fall apart, but want to do some more, take a bit of load off the devs, but not all the way.

The project is in python and its a bunch of different modules that interact with each other.

Developers today do only unit tests, class level, and some very very limited "system" level tests.

I was interested in finding out what other folks do for a mix like this. I want the developers to do more "System" level testing, something where they test their components end to end in the system perspective also, at some scale larger than one operation or two concurrent operations. But I am coming up against the question of what is expected out of this new team in that case. The only things I can think of are scale and a full all-components-together testing, but thats something the QA team will do in any case.

Answer 1

It sounds as if you want to avoid giving QA a build that falls apart right away. One way to address this is to add a smoke test to the development cycle. There are other SQA questions that cover smoke testing, e.g. Smoke testing by Dev team. I won't try to repeat that information here.

One goal of smoke-testing is to encourage developers to deliver QA builds that at least pass a minimum level of quality. Another goal is to avoid wasting the QA team's time. Delegating smoke-testing to the middle QA team might cut down on wasted time for the final QA customer team. I do not think it would encourage developers to improve the quality of the QA builds.

Answer 2

User246 is right, smoke testing by dev team should give QA team more stable product to test.

But I am concerned about your whole approach. You cannot "assure quality by more testing". Better approach (better use of time of all the people involved) is to realize that quality has to be build in, and it is everyone responsibility, not only QA. QA provides information about current status of product quality, but cannot "assure" it if developers do not care about quality.

Better use of intermediate QA team time (best return on time invested) would be to develop automated test for most important functionality and use continuous integration to check for any regressions. Run all those automated test in order of increasing complexity (smoke test first, then happy path of important functionality, then rest). Manual QA testers would test some tricky/specific scenarios for which you do not have automated tests yet.

Answer 3

What we currently use in our continuous integration system is the multiple layered environments, which may also cater for your needs. Let me describe it a bit.

1. Dev Environment - Mostly Dev`s localhost setup in which code gets build and unit tested. What we learned here is that involving a tester early on to help with the scenarios helps a huge deal.
2. System Test - Successful builds delivered to QA(might start here with the new team), deployed and tested early on. Dev implication should be just support here for configuration/bugs. I wouldn`t have the dev team do "More" testing since they are biased on their "little babies".
3. UAT env - to me this mostly looks like the description for your current QA team. Quality of the builds will be greater than System(might want to use a branching system on this one). Also we use this env to let the customers QA Team do their work here.
4. PreProduction - Most accurate replica of the prod environment, mostly used to check the deployment process + quick smoke testing by the UAT/Systest team.
5. Production.

Answer 4

I realize this question is 4 months old, but given what it's about, I figured I'd add my thoughts as you may still be working on putting this team together and determining roles and all.

So it sounds from what you described like the dev team does black-box testing and then the QA team does white-box testing, yes? If that's the case, I see this middle team as being grey-box testers. Where they are considered QA integration testers, but have some access to the low-level particulars of the inputs and outputs of each system.

In that kind of fit, it might look like this: The Devs have their local environments where they code and do black-box unit tests and then build their code into an integrated environment. This "middle" team uses this integrated environment where they are responsible for its data and configuration and perform specific integration test scenarios. They can both run the integrated system in this environment as well as get whatever access to the source details is deemed appropriate. Then systems and configuration that pass integration testing get built into the next environment that the white-box QA testers and/or users test in where they will run full process scenarios.

Hope that makes sense. Good luck!