What expectations should be given to the client with regards to how much time UAT takes and how many issues might be found?

Question

When creating an application on behalf of a client, is there a rule of thumb with regards to how much time User Acceptance Testing might take? For example, a 100 hour project, UAT will take 20% of that time (an extra 20 hours).

Aside from how long UAT might take, how is it best to manage the client's expectations with regards to how many issues may be found during an alpha/beta stage before the client should be happy to sign off on the work and it can go live?

Answer 1

I don't consider this being as easy as using the rule of thumb for neither of your questions as there are several factors that should be taken into consideration.

Let me expand on this.

how much time User Acceptance Testing might take?

• for an e-commerce site I consider 20% for UAT time too much
• for a video streaming platform it may be the right amount
• for a financial / medical project it may be not enough
• for a more complex mobile application it may again not be enough considering you will need to cover more than one device
• are performance & security testing part of UAT?

how many issues may be found during an alpha/beta stage

• expanding on the project types above, again, it depends. Needing to cover multiple devices may uncover multiple issues
• need to take into account the seniority of the developers, their business knowledge level, how versed are they in the domain, technology, etc

Besides all the above, I consider other factors may have a say into this:

• are you working following a Waterfall or Agile methodology?
• how knowledgeable are you, as a tester, in the business domain? Have you worked on similar projects in the past?

As a bottom line, you should when giving an estimate of the sorts, you should take into account:

• how confident are you with that specific product quality? Assuming UAT is not the first time you're doing testing on that project, how major are the issues you expect to find?
• how risky is the industry of your project? For sure you will need to spend more time on a medical one rather than an e-commerce
• how many devices / browsers / platforms do you need to cover? If it's a web project, do you also need to support mobile browsers?

Answer 2

I'm not aware of any good rule of thumb for the effort for carrying out user acceptance testing. As Cosmin said, one factor is the type of application - different types of systems will have different amounts of effort dedicated to UAT. Other factors include who is performing the UAT and their familiarity with the system, the complexity of the changes since the last UAT, and more. In my experience, UAT tends to be a timeboxed activity and is often based on a negotiation between the development organization and the organization accepting the system, and the processes and capabilities of both organizations will factor into the amount of time required for a UAT.

As far as managing client expectations go, UAT should be a formality. Although users may be testing on alpha and beta software, the "real" UAT tends to be something after work is "done". If the rest of the development process was sufficient, the UAT should have feedback but not find any blocking issues. The developing organization's internal development processes, along with any external alpha and beta testing, should find the vast majority of issues before a UAT begins.

Answer 3

As always mentioned testing is context depended , so the time required will change depending on complexity , regression suite, automated or manual , domain expertise requirements etc.

The success of a good test engineer is in justifying why he/sh thinks the time requirment is "20%" , and the justification should be through fact and figures.

Imagine you have to test a policy that has around 100 combination, you should be able to explain following reasons :

1. This is a critical system and all 100 combination should be tested

2. It takes 20hrs to test all the combinations

3. Why can't the process be automated ?

Aside from how long UAT might take, how is it best to manage the client's expectations with regards to how many issues may be found during an alpha/beta stage before the client should be happy to sign off on the work and it can go live?

Clients confidence is not on the number of issues but on the severity and priority of the issues find . The issues might not be just functional issues but also UI design improvements that you feel as important.

Again all this factors depends on context, what you are testing , what are the scope etc.

The best way is to ,

1. Follow TDD , where you start writing test cases in parallel with the development task
2. This allows to have all test scenarios to be ready when you get the build for testing
3. You can save some good amount of time through this as you don't have to wait for the build to start writing test cases

SO in simple words the answer depends on

"Your team size, expertise, complexity of project, feature, and so many factors. In short trust your engineers have good discussion and ask them how much time they would be requiring to gain minimum accepted confidence on the product"