Internal Acceptance Test (not automated): How long should it take?

Question

Given that each test environment is different, is there a general rule of thumb as to how long an internal manual acceptance test should take?

Definitions / clarification: Where I came from, when we got a new feature ready, QA did a quick 30-minute basic acceptance test of core functionality. "If this feature is broken, we are sunk, company closes, we all lose our jobs" was the deciding factor as to what went on this Internal Acceptance Test.

Where I am now, when I arrived here, they would build, and no one re-tested core functionality. So I went to my (very patient) manager & sold him on performing an internal acceptance test once x week. (Since builds are done continuously, every time a new build is pushed was not realistic). So glad I did! Many many critical bugs have been found!

When I designed this test, I tried hard to keep it to 30 mins. As the years have gone by, more and more functionality has been added. We have to test 1 platform on 3 versions, and, as we usually find bugs, this acceptance test has grown to easily take 2 hours on a bad day. We have the usual workflow of customer & QA bugs to verify and I am nervous it is taking so long - it could turn out it never gets done due to length of time to perform.

My question then is: should I be keeping it 30 mins? That was the only example I had to go by at the time.

Thanks in advance.

Answer 1

It sounds like this is the only regression testing of existing functionality you do, is that correct? It sounds like you're finding valuable bugs with your testing, and that your manager appreciates its value.

It also sounds like the other work you have conflicts with performing this testing.

"We have to test 1 platform on 3 versions, and, as we usually find bugs, this acceptance test has grown to easily take 2 hours on a bad day."

This is also an important point: if you're finding and reporting bugs, that takes away from the time where you can be testing. Enough so that you might actually only be spending 30 minutes testing, and the rest of the time reporting the bugs you found.

I don't think the solution to your dilemma necessarily lies in working out how long acceptance testing should take. It sounds like you have more potential work than can be achieved in the time you have: given that test is essentially an information providing role, it's for your stakeholders to decide what information is most valuable to them.

My suggestion would be that you try two things:

1. Bug analysis: Try to analyse the type of critical (as judged by your stakeholders) bugs you're getting - don't worry too much about classifying each individual bug accurately, as that will turn into a massive timesink. Instead, try to just build up a very broad brush picture of where most bugs appear to be coming from, enough that you can go to your manager and say "we appear to be getting a lot of bugs of this particular type, which costs a great deal of test time to find and report. It might well be worth investigating whether there's anything that could be done that would reduce the likelihood of bugs of this nature being introduced in the first place, or reduce the cost of finding and fixing them."
2. Prioritising your tests: Review the areas that you're testing during your acceptance tests. Back when you started, these were all "we're sunk if this breaks" tests. Are those tests still as important, or has what's important to the business changed during that time? Is everything that's crept in since as important? Reviewing these might reveal that there are tests that, while they're nice to have, aren't as important as the other work you might be doing instead.

If you can get the areas that you test ranked in rough order of priority, you may want to suggest that you try timeboxing your acceptance test: once you get to the agreed time limit (including time spent on bug reporting!), you stop. If you've got a prioritised order for the areas you'd normally look at, you know that then you've looked at the areas which are most important to your stakeholders first.

This also makes it easier for them to make a decision about what information is most valuable to them. Given that your time is limited and it's impossible for you to do everything: when you stop, go to your manager and say

• "These areas have not been tested.
• Are you comfortable with us stopping at this point,
• or should we set aside/postpone testing on some of the customer bugs we have in order to continue?
• If so, where on this list should we reach in order for you to feel comfortable about releasing?"

Answer 2

It sounds as if you are referring to a smoke test, so that's how I'll refer to it.

The length of a smoke test depends on your product, the risk aversion of your organization, and the resources available for smoke-testing. In my first QA job, the smoke test could take up one individual's entire morning, but detecting a catastrophic defect during the smoke test could save many multiples of that across the entire QA and development team, so it was worth the investment.

Only you and your team can decide how to trade off smoke test time with the risk of using a build with "if this feature is broken, we are sunk" defects. You should feel free to vary the smoke test depending on which features are bottlenecks for the rest of the app, which features are undergoing the most changes, and perhaps the historical defect rates of the developers working on those features.

Here's something to consider: can you automate parts of the smoke test? At my first QA job, installing the product could take a few hours. We found a way to automate the process, and that made the job of the smoke tester a lot easier without incurring any significant risk.

Answer 3

Time taken to validate a build varies based on various factors.

One way to identify the optimal time that should be taken in your scenario is to put in numbers for the following:

• No. of testers
• No. of features
• Avg. time taken per tester per feature
• (And in your case) No. of versions to be tested

Once you have these numbers, come up with the time taken by 1 tester to test all features for 1 version (assuming the other 2 versions have the same number of features). And then do the required math for the total time required.

In our project we have observed that the total time taken to run an acceptance (smoke) test for about 100 features by 2 testers takes anywhere from 4 - 6 hours.

Answer 4

Full regressive acceptance testing can only, really, be a static value if the application never gets new features. The moment you add a new feature, and that includes support for new web browsers, new operating systems, etc., you need to add tests to cover that new feature. To say that you need to limit it to 30 minutes is unrealistic.

If all you are looking for is a sanity check that the absolute basics of the application, the drop dead "the world is going to end if this doesn't work" kinds of things, then you need to spend time in scope control. When a new feature is added, you need to ask, "Is this a drop dead item"? If it is, the next question is, "Is it worth extending the time for this item or does it supplant an existing item?" If it's the former, then your time gets extended to test the new feature. If it's the latter, than figure out which of the already existing items no longer has a high priority, drop it, and reconfigure your tests.

It comes down to the difference between a full set of regression tests of all features and a "smoke test" of those basic items that are "drop dead". A full regression can probably be limited to weekly. But smoke tests, by their nature, could be run daily or even once per build if they aren't too long.

Answer 5

Any amount of time is OK as long as it's saving you more time than it costs. So if the tests need to run an hour to prevent 30 testers testing a bad build for five hours while a new deployment is created, so be it.

I would seriosly try to invest in automation to get the time down as much as possible. Curating automation can be almost as much hassle as just running manual tests, but normally it's a big win.

If your product is impossible to automate it's probably got architectural issues. Try to talk the team into making new features automation testable by design.

Answer 6

keep it short enough so it doesn't disturb other testing activities (much), that's what I usually try to get to.