Selenium Webdriver - handling error messages that are shown to user

Question

I've made an automated test case that creates a user inside an app. It's pretty simple, the test enter username, email, password and presses OK.

But how should I handle scenarios if user enters an username or email that is already taken? The app shows a message to the user if username or email is taken but I don't know how I should handle such scenarios where Selenium tests enter info that cannot be used.

Should I check if a message pops up after clicking "OK" button? If yes, should the test show a note in test report and end the test or try entering different values?

Another idea - should I have a separate test scenario if user enters username or email that is taken?

I really need some ideas that would be optimal.

Answer 1

It depends

The best way to handle your scenario depends on the way your application handles the scenario.

• If the app notifies the user that username/email is taken as soon as they move to the next field: In this scenario, you should have a separate test for each possible situation. The minimal set of tests I'd use are:
  • Everything works - the username and email are unique and meet whatever other criteria is used.
  • Username is taken - for this I'd take the test as far as checking that for an already taken username the expected error text is displayed.
  • Email is taken - again, the test would be that for an already taken email, the expected error text is displayed.
  • User chooses unique username after being notified of already taken username - for this scenario, I'd want to verify that the error message is cleared once a unique username is selected.
  • User chooses unique email after being notified of already taken email - again, the verification is that the error message is cleared once a unique email is selected.
• If the app does not notify the user until after they submit: In this situation, the way I'd test would depend on the way the notifications work and how they display. In general, if notifications call out each invalid field, I'd have a test per field. If they are less helpful (which I've seen entirely too many times), then a single test of invalid values might suffice.

In general it's better to test each potential path or scenario separately. That said, there is a point where it's better to simply stop because the cost of time to create and maintain automated tests exceeds the value they provide. Theoretically you could cycle through duplicate usernames or emails an infinite number of times before the user finds one that is unique. In practice, there's probably going to be a point where a user said "It's not worth it" and stops trying.

If you have sufficient logging around the app, you can start with the basic scenarios and use the logging to determine the average number of tries it takes for a user to select a unique value, the maximum number of tries, and any other statistical values you care to use. That will allow you to make an informed decision on how far into the "User chooses duplicate, user chooses another duplicate, user chooses another duplicate..." loop you need to go.

Security implications: You should be aware that telling the user they need to choose another username or email is giving hackers the ability to find usernames/email addresses that are valid for your app, and consider some of the risks in tightening the number of times someone can enter a duplicate vs the risks in allowing someone to continue indefinitely.

Answer 2

General approach to simulate user interaction is invoke some control and watch if the UI stated has changed. So applying this basic conception to exceptional use-cases, the test could input some "wrong" data and wait for error message to appear

Regarding the test composition this is quite specific area. I mean each time there are certain factors which make the particular case different from another. So to take decision on whether to cover occupied email and occupied user name in a single test you might want to consider the following (all about test maintenance convenience):

1. What is the probability of changing uniqueness requirements in future. Is it possible for example that login name uniqueness requirement will be decommissioned or some new user property will be added to uniqueness requirement.
2. How long does it take to set up precondition for the test that operate on the page where you test uniqueness
3. Do you expect the situation when you'll need to execute uniqueness test for one field skipping the test of another (for example for some precise bug-fix test)
4. If there is designed way to accept error message so that the user can proceed with next attempt

If to talk about my personal preferences, I would rather distinguish the test for each of fields. And each of the tests I would implement parameterized so that I could test a single field for different classes of "errors".