what should I do if my stability tests cycle is longer than the release cycle ?

Question

Our long term stability test cycle takes about two weeks to complete, during this time another release (sometimes two) is being released. What should I do ?

The test cycle can't be shortened, and we can't run two releases concurrently.

Answer 1

Your options here would be controlled by dependencies & overlaps between the two test cycles.

Analyze them first both short & long term -- resources,test infrastructure,test results,business impact

Then,as you mentioned that you can not run two releases concurrently, try and spot overlaps,see what work could be done which would help both the test cycles.

Is there any ? If yes - great,you can support that.

if not(and for not overlapping areas of work) try decoupling resources,activities or even look at outsourcing to another team

and as a last resort,prioritize and come up with a suggestion list for the business/project leadership to take a call,on which items needs to be dropped ?

being fearless and breaking the news of inability to sustain is a better approach rather than reeling under the pressure of both deliverables and failing in both...

Answer 2

Ok so if I understand correctly here is your scenario:

• A. you have a long period of testing say 2 weeks and that cannot be shortened.
• B. you have to release multiple times within that 2 week period say, once a week.
• C. you cannot run two releases concurrently.

Time for some tough love, something has to give you either need to :

1. Identify a smaller subset of tests to reduce what you are testing, but this violates A.
2. Reduce the number of releases so you release only once every 2 weeks but this violates B.
3. Run two releases concurrently, which violates C.
4. Use automation to reduce the execution time, which again violates A.

So, simply put, one of the three rules will need to be broken, otherwise you simply have to accept the fact that your testing will never be completed on any given release.

Answer 3

Your question as stated is not a solvable problem. Assuming this is the desperate cry for help of the tester trapped at the end of a waterfall process the flip answer is "get more agile."

Of course the usual problem in this situation is that the QA team has very little leverage beyond "Test Sign Off". Worse, if bugs are released the QA team is called on the carpet.

Here are some leverage points to consider.

• Take your problem statement and turn it around. Ask "What are we really trying to accomplish?". This question has a lot of power to get to the heart of problems that seem intractable.
• Ask why five times.
• Ship with less testing as an experiment. Have a way to "pull the plug" if things go wrong.
• Tell the Dev team that the QA team no longer does "test sign off". The Devs decide when a feature is stable enough to ship. The QA team is on the hook to provide the measurements and data to help them decide.
• Pick one way to improve the overall process and get one developer to try it out for a release cycle.
• Make one thing a little better every day. Sometimes chipping away at a hard problem will cause it to unexpectedly crumble.
• Keep a positive attitude. Focus on what the team is doing well as often as you can.
• Repetition. Pick a catch phrase for one improvement you want to see ("testable by design" was a good one for me) and repeat it until it catches on.
• Make sure you aren't part of the problem. Don't say "We can't ...", "We'll never ...", "Bob is an idiot.", etc. Instead consider saying "We don't yet ...", "We haven't started ...", "Bob isn't wrong about ..."
• Start saying "and" when you mean "but". Say "we" when you mean "they". "We can't complete a testing cycle in less that two weeks AND we need to ship once a week."

Solving seemingly intractable problems is all about finding some leverage. Finding leverage is a classical exercise in thinking outside the box. You have to tap into your passion, patience and be persistent. Look for the small wins and celebrate them to get some momentum.

Answer 4

It would depend on what your goal for stability is, do you need the 2 weeks to be able to generate errors that have been reported or has it "always been done" this way?

I had to maintain a stability environment in a previous job, the rationale for this was because we needed to run load and try to find defects that often only showed up after X# of days. Even though we still had releases going out the stability environment had to run for its time, although we would only update after we reached the expected time then did the update.

If you have to run your test for this amount of time, there probably is not much you can do, if you can shorten it as suggested then discuss with the people who set up the test what you can do to make it realistic and fit within a time you are more comfortable with.