3

We develop 3 apps in total, all on iOS and I'm running into serious time issues regarding the iOS Beta Releases (current release is iOS 11.2.5 Beta 2). We want to perform a full regression on all our apps whenever an iOS Beta release comes out. This takes the best part of a day. Can't automate as it's to be performed on the version from the App Store.

The issue is that the Beta releases can come at any time, sometimes once every few weeks or sometimes twice in a week (like the one just gone). Does anyone else have issues with this? Is there merit in performing full regressions for each iOS Beta Release considering how much time it eats into our testing? Is there anything I can do to speed things up or any advice on what to check in iOS Beta Releases (it's generally just bug fixes although Apple don't seem to give a list)?

Thanks for reading, I'd appreciate any advice you have.

Improve this question
2
  • Why do you test for betas in first place? How many user of your apps install betas of iOS?
    – Embedded
    Commented Dec 21, 2017 at 14:29
  • We had an issue back in iOS 8 I think where a general iOS release broke our apps and we found that it could have been spotted in early Beta testing.
    – JordiLaForge
    Commented Dec 21, 2017 at 15:20

1 Answer 1

Reset to default
2

What you have here is a classic case of whether the ROI makes the effort worthwhile. You can find recent release notes on Wikipedia, as well as the Apple developer library.

Considering how many releases of iOS there have been since iOS 8 first appeared, one critical issue is not a bad track record.

Unless your apps deal with financial or other critical, time-sensitive data, I would suggest that your goal be to start by reviewing release notes as soon as they become available, looking for anything that seems likely to impact one of your apps. I'd be particularly concerned with bug fixes and known issues, since they're the most likely places to interact poorly with other software.

Regression against the operating system shouldn't need to happen unless you find something that could potentially interact.

Then, if a problem does happen, you put a big flashy notice on your website that there is a problem and you're working to fix it; and you flag your app(s) in the app store to be incompatible with that version until you've fixed the problem.

If your apps are mission-critical enough that you can't afford any downtime, then your organization is going to have to afford the time cost of regression against each new iOS version. If it isn't, you don't need to do the regression as long as you sanity-check the release notes.

Improve this answer
1
  • Thanks for the comment. It's just a shame that Apple isn't more transparent about their releases. We've decided to ignore the 1st Beta build of each release as they've only ever released after one beta build once so that's one less regression per release. The team really wants each build to be tested and they've allotted time specifically for it. For now, I'll still be regression testing each beta. It's an arduous task and I fear errors will slip through the more it's performed by the same person. I'm putting together data from all previous beta releases and will explore automation again.
    – JordiLaForge
    Commented Dec 22, 2017 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.