Are Exhaustive "Brute Force" UI Tests Worth It?

Question

Lately I've been asked to write some automated GUI tests of the following style:

• Open the application
• Open ALL dialog boxes
• In each dialog check for ...

Are these types of tests worthwhile? What are some benefits of these tests? In some cases they can be very simple to write, but in others they require extensive knowledge of many unrelated parts of an application. As well, they can take a long time to finish, finding issues or not. Sometimes I feel these tests just aren't worth it, but I could be wrong. Any thoughts?

Answer 1

To answer the question "are they worthwhile?", you need to explore:

• How expensive (in terms of time and money) are they to create, maintain, and execute?
• What value are these tests providing? Are they finding bugs? Providing confidence?
• Are there other, less expensive, ways to provide similar value?

I once worked at a software company which built desktop tools used by large-enterprise DBAs. We had an intern whose responsibility it was to thoroughly examine the UI and measure it against our standards for UI object size, placement, spacing, font selection, etc. He basically performed all of the steps you indicated, and more. We were eventually able to automate some of that (it was in the early days of test automation), but not all.

In some shops it would have not been a worthwhile exercise. In ours, it was considered money well spent. For my company, complete UI consistency was considered an important attribute for our intended market.

Answer 2

Answering your question requires knowing why you were asked to write those tests. Unlike solving applied math problems, we usually do not write automated UI tests for the sheer pleasure or the intellectual challenge. I assume you were asked to write them, in that style, for a reason. If there is a reason to believe the UI is particularly buggy and that it is easier to test the UI automatically than manually, it may make sense to write them.

Sometimes testers resort to automatic UI testing for the wrong reasons. The reasoning usually goes something like this: the UI is ultimately the interface the user will interact with, bugs in the business logic will not necessarily be triggered by how the UI uses it, and when you test the UI, you test the business logic at the same time. However, a well-architected application will separate the UI from the business logic. It is usually better to use API-level tests for the business logic; the tests will be faster and better able to take advantage of API-level diagnostics that are not available in the UI. They will also be insulated from UI changes (although of course they will not be insulated from API changes).

You have been in SQA for a while, so I suspect you already know this, but I will mention it for completeness. A line of reasoning promoted by some tool vendors and UI automation evangelists goes like this: automation is hard because it requires programming, but these days there are great capture/playback tools that enable creating automated UI test suites quickly with a minimum of programming knowledge. You did not say what kind of technology you would use to write those tests, but if you are considering using capture/playback, you may want to read some other questions on that subject in this forum.

Answer 3

I agree with Aniket, this kind of exhaustive UI test is not worth it. Another thing to consider:

Awhile back, I inherited a manual test case plan from someone who left the company. He had been doing things like checking the list sorting behavior in every area of the app. He wasn't a programmer and didn't understand that the code behind the dropdown box for the sorting mechanism was the same each time. He was over-testing. This is the kind of thing you'd run into if your test plan called for interacting with every widget possible.

Remember you're doing functional testing of the behavior of the application under test, not testing whichever UI framework the developers chose to use. Chances are you'll find any UI-specific bugs when you are executing your functional tests anyway.

Answer 4

IMHO, no.

Here's why:

1. Depending upon your UI, cost of implementing will probably be high. Don't forget to sum the cost of maintaining it when your UI changes in the next release and so on. This itself should warrant second thoughts.

2. UI tests can be flaky. In my experience accuracy of UI test cases is pretty low. I'd imagine this to get worse with brute forcing.

3. If you have 10 elements are there going to be 10! iterations?!

4. Time and resources required to execute these TCs and then investigate failures might outweigh the benefits.

5. If you do implement and execute something like this, will you be 100% sure that your UI is near perfect? :-) I guess not. There's your answer.

Hope that helps.

Answer 5

I believe that automated testing scenarios need to be build around real life use cases, so meaningfull testing script should be close to what user typically would do

If in your system it is a typical for a usual user to open all forms and fill all checkboxes, then, yes, otherwise - it is useless. Even if this testing will find some bugs and deelopers will spend some of their time to fix them for a scenario which will never occur, wouldnt it be better for developers to spend this time for creating a new features for their users?