How do you handle your automation deadlines if Appium/iOS/Android versions keeps updating and these updates have issues?

Do you test with the old versions for your deadlines or test with the updated versions? But what if the updated versions gives errors and you end up not being able to run your tests?

How long do you guys wait until you upgrade to the new versions to run your automation tests? How do you handle this period when there are changes to the Appium or iOS or Android versions?

  • Do you have list of supported versions of third party software? – Peter M. Oct 12 '15 at 21:08

Testing with old versions makes little sense. Only do it to familiarize QA team with new features (if changes are not radical), and develop testing protocols and tools.

To deal with changes in system to be released, we use "soft freeze" and "hard freeze".

During "soft freeze" changes in released software need to be approved by QA (and are evaluated for likelihood of breaking system under test), and/or chances to adding new unknown bugs. It depends on developer, affected subsystem, and complexity of the change.

During 'hard freeze" we run all tests again, and no changes in released software are allowed. Most patches requested during "hard freeze" are delayed for after the release. Or if decision is made that urgent patch has to be made before release to "hard frozen" system, there is understanding that defects might sneak in, which QA was not able to detect, because system under test was changed.

Idea is that QA cannot "assure" that release has no (unknown) bugs, but provide best information to management about the risk of releasing software with bugs.

Decision to release on a given date (or postponing the release to allow for more testing, if necessary) is management decision, and role of QA is to approximate the risk. Fixing the bugs is responsibility of developers, and evaluation the risk/value is management decision.

We all work as a team (developers, QA, management) to deliver the best (most features) and most reliable system in the time and personnel available. If you do not have such level of trust, and after botched release there is fingerpointing, shifting the blame, and searching for sacrificial goat to blame, everyone will start padding the estimates and speed of the development will decrease. Let management to manage the project, but QA should provide best info about the status of the project it can.

Regarding versions of the third party software, you need to decide which versions you support (with the limited resources you have), which versions you do not, and which versions will get obsoleted in some defined timeframe. Do you have control over which version your user use? If not, some users will have to deal with bugs limited to their version - and again, your management has to decide what threshold of such bugs is acceptable. So you may not support most recent "bleeding edge" version (if you decide so), only 2-3 previous "stable" versions. Then you need to detect which version user uses, and issue proper message.

Again, your role is to investigate the risk involved with different versions, and provide info to management to make business decisions and allocate resources as makes business sense.

  • Thanks Peter. Good to know about soft freeze and hard freeze. During soft freeze when you play with the new upgrades and you find there are issues, do you decide to revert the testing back to the old versions to continue? Or do you wait until there's fixes for the new versions and test again? So copies of old versions are always maintained in case of need to rollback to previous stable versions? – seleniumappiumnewbie Oct 12 '15 at 17:57
  • Why would you revert a feature planned for release? Bugs needs to be fixed. Only decision is if we want to fix in this coming release, or release with a known bug and fix later. It depends how close are we to hard freeze, how annoying is the bug (affecting just few customers? All), how dangerous the patch is, etc. basic triage. – Peter M. Oct 12 '15 at 18:12
  • Sorry if I confused you. No I'm not talking about reverting a planned feature for a release. I'm talking about outside parties upgrades like an Apple IOS upgrade, Android upgrade, Xcode upgrade, Appium upgrade that can affect if your test cases pass/fail since sometimes the new operating system upgrade and the new tool version is not compatible until they release a new version of the tool. – seleniumappiumnewbie Oct 12 '15 at 18:19
  • For example, If I ran my test suite with ios 8.4 and Appium 1.4.8 and it passed. Then around our deadline time, Apple releases ios 9.0 and Appium released 1.4.11, do I upgrade to ios 9.0 and Appium 1.4.11 to retest my test suite before launch? Or do I retest the test suite with the old ios 8.4 and Appium 1.4.8? If I upgrade to ios 9.0 and Appium 1.4.11, it may have issues causing my tests not to pass, but it could be because of the recent changes and not because of our product app. Do you extend the deadline then to test for the new ios? – seleniumappiumnewbie Oct 12 '15 at 18:32
  • I added a blurb about third party sw, but it is the same: evaluate the risk, let managers make business decisions based on best info available. – Peter M. Oct 20 '15 at 16:17

It is important to note the difference between system under test versus test system failures.
It is also important to assess this risk as early as possible during test planning.

General rule is to test in the known functional test environment that is as up-to-date as possible. If the risk materializes that new relevant 3rd party software releases during testing, try it out quickly.
In case of large problems roll back and escalate.

Appium you control yourself so never should break you process.

For Android usually you will get away with proper bug registration and possibly a minor release later down the line as adoption is slower (albeit at the upper end first).

For iOS the situation is a lot worse as Apple enforces early adoption through company policy and update options. Here best have your developers eye pre-releases and possibly try out a beta ahead of the main release.

Speaking Murphy this should happen just about 10 out of 10.

  • Thanks. Yeah I image it would be harder with iOS when there are upgrades. – seleniumappiumnewbie Oct 23 '15 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.