How is enough to automate validation of a form (negative test cases)

Question

Let's say, I have a form has 4 fields (all mandatory with validation messages): Name, EMail, Address, Phone. And a Submit button. Let's assume I have some test cases as below:

TC1: All fields empty and click Submit -> ValMess = Please enter all mandatory fields

TC2: Fill in all fields but Name -> ValMess = Please enter your Name

TC3: Fill in all fields but Email -> ValMess = Please enter your Email

TC4: Fill in all fields but Address -> ValMess = Please enter your Address

TC5: Fill in all fields but Phone -> ValMess = Please enter your Phone

So when I write automation tests, I just automate TC1 to check validation message. Is it necessary to automate all test cases to check validation messages for all scenarios? Or what is the best way to automate in this case to ensure the correctness?

EDIT: Or in another word: Should we automate all negative test cases? And how is enough? Because if we automate all negative test cases, then there would be a lot of code to be done.

Answer 1

My approach, learned from doing this at several companies in different industries, is:

Have a smoke test first (initial page load usually) and if that fails, skip the others.

After that I have 1 sad test per form (frequently per page but not always).

Each sad test will fill out all the fields (using a common helper that is also used by other sad and the happy path). Then I blank out 1 field and submit the form. I then make sure the error message appears and all the field values were retained from the original post. This means multiple assertions for these tests and this is a case where I consciously break the 'one assertion per test' rule or guideline. The reason is ultimately speed - when driving a browser it is unwise to go to a page 30 times in a row for every page form being tested in a reasonable and reliable manner. I call this the one 'group assertion' per UI test exception where all the group members (the assertions) are checking simple field values being retained by the form re-display.

Finally I have the happy path for the perfect typist with all correct data entered first time.

This doesn't test all the error messages and combinations but I prefer to create jasmine or protractor tests for that sort of testing.

Answer 2

You cannot automate all negative tests. How can you?

• There are infinite ways of doing something wrong. It will take forever to automate negative test scenarios for one given scenario.

Having said it, we do need to automate selected negative test scenarios.

• We can partition test scenarios into different categories and automate one negative test scenario from each category.

How do we know when is enough?

• It depends on your budget, your allocated effort and the subject under test. if the subject under test is life-critical, it will demand more thorough testing.

Answer 3

There's no need to automate every possible negative test case. Something like that will only drive up the time it takes to run your tests, while bringing less and less business value each time (think diminishing returns.)

While this isn't exactly a problem in something as simple as what you described (a form with four fields) imagine if the application grew to have more, or you're working on an application that requires significantly more test cases.

So I guess the bottom line answer here really is to automate what your business needs.

Answer 4

In your provided example, there are only four test cases necessary.

One for each mandatory field to test for empty/invalid. In each scenario all other mandatory fields would be filled. The only exception to this would be if the acceptance criteria stipulated that more than one error at a time would be reported.

It's that plain and simple.

Answer 5

I would also add the following Test Cases

• no Address or Phone#
• No Email, Address or Phone#
• No Email, no Phone#
• only Email filled in.
• click Submit after Validation Message Shows
• Double Click the Submit Button while the fields are complete
• Double Click the Submit Button while the Fields are empty
• Triple Click the Submit Button while the fields are complete
• Triple Click the Submit Button while the fields are empty
• Hit Refresh (if web application) while the validation message is present.

for the sake of Regression testing, you should automate as many tests as you can, that way every time a change is made to the application you can test all aspects of the application, even the ones that you don't think are going to be affected by the change.

Answer 6

I would test exhaustive negative input scenarios using unit tests , not with UI tests.Its expensive.

On UI level, I will design & test main(very few) happy & Sad tests only.

The reason being, unit tests are far less expensive than UI tests in terms of design, setup & execution.