The best way to handle your scenario depends on the way your application handles the scenario.
- If the app notifies the user that username/email is taken as soon as they move to the next field: In this scenario, you should have a separate test for each possible situation. The minimal set of tests I'd use are:
- Everything works - the username and email are unique and meet whatever other criteria is used.
- Username is taken - for this I'd take the test as far as checking that for an already taken username the expected error text is displayed.
- Email is taken - again, the test would be that for an already taken email, the expected error text is displayed.
- User chooses unique username after being notified of already taken username - for this scenario, I'd want to verify that the error message is cleared once a unique username is selected.
- User chooses unique email after being notified of already taken email - again, the verification is that the error message is cleared once a unique email is selected.
- If the app does not notify the user until after they submit: In this situation, the way I'd test would depend on the way the notifications work and how they display. In general, if notifications call out each invalid field, I'd have a test per field. If they are less helpful (which I've seen entirely too many times), then a single test of invalid values might suffice.
In general it's better to test each potential path or scenario separately. That said, there is a point where it's better to simply stop because the cost of time to create and maintain automated tests exceeds the value they provide. Theoretically you could cycle through duplicate usernames or emails an infinite number of times before the user finds one that is unique. In practice, there's probably going to be a point where a user said "It's not worth it" and stops trying.
If you have sufficient logging around the app, you can start with the basic scenarios and use the logging to determine the average number of tries it takes for a user to select a unique value, the maximum number of tries, and any other statistical values you care to use. That will allow you to make an informed decision on how far into the "User chooses duplicate, user chooses another duplicate, user chooses another duplicate..." loop you need to go.
Security implications: You should be aware that telling the user they need to choose another username or email is giving hackers the ability to find usernames/email addresses that are valid for your app, and consider some of the risks in tightening the number of times someone can enter a duplicate vs the risks in allowing someone to continue indefinitely.