Is UAT really UAT?

Question

I have noticed that many development teams automate UAT with many test cases. It's things that users care about, but the tests are not done by real users. For example, automation will test whether this is true or not: "When user password is incorrect, the show error message". This is an excellent practice, but I don't understand why this is called UAT?

The definition of UAT includes statements like "... software is tested in the real world by its intended audience" and "In UAT, users are given the opportunity to interact with the software before its official release"

Even the name itself says it User Acceptance Testing (The user has to accept it, not a developer on behalf of the user)

This is very different from the UAT tests that I see which are often:

• Written and maintained by developers (not end users)
• Automated in a "lab" environment, not a real one

Again, I'm not saying that there is anything wrong with it, it's just confusing to me that it's called UAT. The only explanation that I could come up with on my own is that real users are busy, bored, and uninterested in doing real UAT, so developers have to fill the void by creating virtual users that test the system. Would a better way to think of it be pre-UAT, or simulated-UAT?

The related question is, how is "real" UAT possible in a CI/CD environment? The automation and short cycles seems to make real UAT infeasible. For modern CI/CD styled SaaS applications does anyone really do UAT with real humans anymore?

What terminology do QA people use when talking about "real" UAT vs "simulated" UAT?

Am I understanding UAT, its (mis)use by necessity, and its limits correctly?

Answer 1

I don't fully agree with your definitions of UAT. "Real world" is a bit vague and could imply "production", which isn't the case - UAT could be performed in a separate, isolated environment. UAT could also happen after release, but before a new user accepts and starts using the software or a particular feature of the software. The key part of defining UAT is that it is performed by people with a deep understanding of the user requirements and how the system will be used in its intended context and can evaluate its suitability for use in that context.

The type of software delivery model matters here. Organizations building custom-made software for a particular set of users are in a different situation than organizations building software for sale in the marketplace. When building custom-made software, the development teams tend to have a deep understanding of the customer and user requirements. When building software for a marketplace, the development teams need to keep their eye toward a broad set of requirements that satisfy many prospective customers' and users' needs, and sometimes those requirements will conflict, so you'll have to explicitly not satisfy someone's needs.

In both cases, I would look at who designed, reviewed, and approved the test cases. If the test cases were at least reviewed and approved by someone in the capacity of a user, then you may be able to consider the execution of those test cases to be acceptable as a UAT. A development organization could build a customer's user acceptance test suite into their test suite, run those tests, and supply reporting on test execution status to customers as part of their development processes.

There are also organizations that provide validation-as-a-service and UAT-as-a-service. The organization buying the software could contract a different organization to develop a deep understanding of their needs and carry out UAT on their behalf. Sometimes, a large software vendor may have a professional services team that could be hired to carry out these tasks or it could be third-party Although they aren't usually part of the development organization, I don't see why they couldn't be, barring some requirements in regulated industries for independence in verification and validation activities.

The last piece of the puzzle is the timing of the UAT. It's perfectly possible to carry out UAT activities when you are performing Continuous Deployment. Feature flags can help to control when users get access to functionality and can allow for deployment of functionality to production and enabling it for users after they have completed their UAT activities and approved the functionality for use. There are several types of feature flags depending on your specific needs.

Answer 2

At my place of employment, we have an environment called UAT where the only testing done is by me (QA) to check that the environment is stable once we have deployed there what I have already tested in a lower environment. This is simply because the business will not provide anyone to do UAT testing. Effectively, we have an interim environment just for the sake of it. IMO, UAT is testing that is done by users or someone selected/trained to do user testing (the processes that real users perform on the system on a daily basis), regardless of the environment that it is done in.

Answer 3

The related question is, how is "real" UAT possible in a CI/CD environment? The automation and short cycles seems to make real UAT infeasible. For modern CI/CD styled SaaS applications does anyone really do UAT with real humans anymore?

You are right that UAT in its traditional way is not possible in a CI/CD environment, but you can get around it by using split delivery methods, like canary release or A/B testing, and by getting a real feedback from a small subset of your real customers. Obviously there is a risk involved in using this method, but on the other hand you get fast and reliable feedback assuming you setup your monitoring and feedback properly.

Answer 4

In my opinion, UAT testing is all about identifying and meeting the benchmarks of product success. Besides, in one way or another, applications even when tested by developers, is a form of UAT testing since developers in their real-time environment have the knowledge of how authentic user could think or interact with an application.

Thus, taking a product to different developers within a defined setting allows for generating reports that contain perspectives. All in all, be it simulated testing or real UAT testing, I believe it allows testers to have enough exposure to user acceptance. Though every user could be different in their approach to using a digital product, UAT can still help testers to gain enough vision.