Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
How do you handle your automation deadlines if Appium/iOS/Android versions keeps updating and these updates have issues?
Do you test with the old versions for your deadlines or test with the updated versions? But what if the updated versions gives errors and you end up not being able to run your tests?
How long do you guys wait until you upgrade to the new versions to run your automation tests? How do you handle this period when there are changes to the Appium or iOS or Android versions?
Testing with old versions makes little sense. Only do it to familiarize QA team with new features (if changes are not radical), and develop testing protocols and tools.
To deal with changes in system to be released, we use "soft freeze" and "hard freeze".
During "soft freeze" changes in released software need to be approved by QA (and are evaluated for likelihood of breaking system under test), and/or chances to adding new unknown bugs. It depends on developer, affected subsystem, and complexity of the change.
During 'hard freeze" we run all tests again, and no changes in released software are allowed. Most patches requested during "hard freeze" are delayed for after the release. Or if decision is made that urgent patch has to be made before release to "hard frozen" system, there is understanding that defects might sneak in, which QA was not able to detect, because system under test was changed.
Idea is that QA cannot "assure" that release has no (unknown) bugs, but provide best information to management about the risk of releasing software with bugs.
Decision to release on a given date (or postponing the release to allow for more testing, if necessary) is management decision, and role of QA is to approximate the risk. Fixing the bugs is responsibility of developers, and evaluation the risk/value is management decision.
We all work as a team (developers, QA, management) to deliver the best (most features) and most reliable system in the time and personnel available. If you do not have such level of trust, and after botched release there is fingerpointing, shifting the blame, and searching for sacrificial goat to blame, everyone will start padding the estimates and speed of the development will decrease. Let management to manage the project, but QA should provide best info about the status of the project it can.
Regarding versions of the third party software, you need to decide which versions you support (with the limited resources you have), which versions you do not, and which versions will get obsoleted in some defined timeframe. Do you have control over which version your user use? If not, some users will have to deal with bugs limited to their version - and again, your management has to decide what threshold of such bugs is acceptable. So you may not support most recent "bleeding edge" version (if you decide so), only 2-3 previous "stable" versions. Then you need to detect which version user uses, and issue proper message.
Again, your role is to investigate the risk involved with different versions, and provide info to management to make business decisions and allocate resources as makes business sense.
It is important to note the difference between system under test versus test system failures.
It is also important to assess this risk as early as possible during test planning.
General rule is to test in the known functional test environment that is as up-to-date as possible. If the risk materializes that new relevant 3rd party software releases during testing, try it out quickly.
In case of large problems roll back and escalate.
Appium you control yourself so never should break you process.
For Android usually you will get away with proper bug registration and possibly a minor release later down the line as adoption is slower (albeit at the upper end first).
For iOS the situation is a lot worse as Apple enforces early adoption through company policy and update options. Here best have your developers eye pre-releases and possibly try out a beta ahead of the main release.
Speaking Murphy this should happen just about 10 out of 10.
