It sounds like this is the only regression testing of existing functionality you do, is that correct? It sounds like you're finding valuable bugs with your testing, and that your manager appreciates its value.
It also sounds like the other work you have conflicts with performing this testing.
"We have to test 1 platform on 3 versions, and, as we usually find
bugs, this acceptance test has grown to easily take 2 hours on a bad
This is also an important point: if you're finding and reporting bugs, that takes away from the time where you can be testing. Enough so that you might actually only be spending 30 minutes testing, and the rest of the time reporting the bugs you found.
I don't think the solution to your dilemma necessarily lies in working out how long acceptance testing should take. It sounds like you have more potential work than can be achieved in the time you have: given that test is essentially an information providing role, it's for your stakeholders to decide what information is most valuable to them.
My suggestion would be that you try two things:
- Bug analysis: Try to analyse the type of critical (as judged by your stakeholders) bugs you're getting - don't worry too much about classifying each individual bug accurately, as that will turn into a massive timesink. Instead, try to just build up a very broad brush picture of where most bugs appear to be coming from, enough that you can go to your manager and say "we appear to be getting a lot of bugs of this particular type, which costs a great deal of test time to find and report. It might well be worth investigating whether there's anything that could be done that would reduce the likelihood of bugs of this nature being introduced in the first place, or reduce the cost of finding and fixing them."
- Prioritising your tests: Review the areas that you're testing during your acceptance tests. Back when you started, these were all "we're sunk if this breaks" tests. Are those tests still as important, or has what's important to the business changed during that time? Is everything that's crept in since as important? Reviewing these might reveal that there are tests that, while they're nice to have, aren't as important as the other work you might be doing instead.
If you can get the areas that you test ranked in rough order of priority, you may want to suggest that you try timeboxing your acceptance test: once you get to the agreed time limit (including time spent on bug reporting!), you stop. If you've got a prioritised order for the areas you'd normally look at, you know that then you've looked at the areas which are most important to your stakeholders first.
This also makes it easier for them to make a decision about what information is most valuable to them. Given that your time is limited and it's impossible for you to do everything: when you stop, go to your manager and say
- "These areas have not been tested.
- Are you comfortable with us stopping at this point,
- or should we set aside/postpone testing on some of the customer bugs we have in order to continue?
- If so, where on this list should we reach in order for you to feel comfortable about releasing?"