How to investigate a bug of mandatory fields not being entered?

Question

I'm working on a questionnaire which contains mandatory questions such as Date of Birth (three different drop downs are provided), plus a check box and some Yes/No type questions.

Without answering these questions, users are not supposed to be allowed to move to the checkout page, however, orders are being created by live users, yet these order have DOB null and other mandatory answers are also null (unanswered).

When I asked live users how they got past the restrictions, they just reply that they answered all the questions on the form.

What is the other way to test this questionnaire?

Answer 1

If users are reporting that they did provide the values but your view of the system is saying they didn't I would:

• Try using values for future dates
• Try using incognito mode if possible
• Try using a date over 120 years ago
• Observe how form works with valid data
• Try using enter vs. using a submit button
• Try using invalid characters or control keys
• Test with an always invalid date, e.g. 02/30/1969
• Search Application and Web Server logs for errors
• Try using different browsers, Chrome, IE, Safari, Firefox
• Try providing only two out of three values (various combos)
• Make sure both views are using the same database connection
• Try correcting an error but make a different one and submit again
• Open dev tools and look for errors being logged on the network tab
• Test with a valid date using European format, e.g. test with 31/07/1968
• If you have multiple application or database servers, see if one is the cause
• Check the database while testing to see if entries are being created anyway
• Try disabling and enabling javascript and watch network tab while using the form
• Add a bunch of print statements to the source code to help debug what is going on
• Try associating records with issues with web/application logs to see common factors

Note that many of these test, e.g. future date are not necessarily reflecting users intentionally entering such data but more that they may not understand the UI and be accidentally entering such invalid data (or they click on wrong field, etc, etc.) or the UI itself is misbehaving without that being obvious. Having these things tested can often uncover such issues.

Answer 2

I suspect the server-side misses mandatory field validation. Normally when the form is being validated on the page to have all the mandatory fields filled, JavaScript is used. However it might happen that there is

• either a defect in JavaScript code or page elements code

for example the button might be enabled by default and JS sets it disabled once loaded. hence if there are some issues with JS loading, your button will never be disabled

• or JS fails in some browsers due to incompatibility reasons
• or JS is switched off at all in browser security preferences.

Hence I would test if there is server-side validation in your system. Having the input validated only on client side is a sort of bad practice that shows the green light to hackers and gives different ways to introduce inconsistency between front-end and back-end.

Answer 3

This seems to be the DB related issue. The values entered by the live users may not get updated properly in the Database. To test this connect your Database with the staging environment and enter the mandatory field in the website and see the XML generated with the fields getting mapped with the fields in the database and see the values are getting passed in the Database.

Answer 4

Is there any commonality that you can identify for these end users whos data isn't getting captured (such as time of day, location, browser, server, etc.)? Based on your info I am guessing it's all or nothing. I don't know your application, but is it hosted on multiple servers and you might have a different version of your application on different servers that does data validation differently (or is broken on one version where that data isn't being captured properly)?

Depending on how you're doing your field validation, (as Alexy R indicated in his response to you), if you're doing your validation on the client end and using Javascript, they may have Javascript turned off, and so they're not actually doing any validation for you and perhaps without that Javascript running, the data getting sent to your server is not in a format that's expected. Is there anything in your server (or DB) logs indicating an error on those database inserts when you don't get the expected data?

In general, relying on any sort of validation being done on the client side is not a good idea. All data validation should be done on the server side to help insure that data isn't being messed with and that your validation is doing what it should be doing. Also, you don't want a hacker trying to do SQL injection (or other bad data inputs), so you should be doing server-side validation and format checking on every input you're getting back.

Answer 5

In order to find the bug, you may want to insert code into the page that saves out the entry points as plaintext without subjecting them to data types, then submits them internally into the fields as intended. Then have some post-code flag any entry with a null date and forward you the original data they entered. It should be easy to see the pattern causing the failures there, which will give you a starting point.

Answer 6

1. Use appropriate html5 input types for all inputs that fit those types.

2. Reject the submission and notify the user if there are missing required fields.

3. Redifine the input data to a string (i.e. whatever = whatever.toString() ) and use regex to ensure the submitted input data matches the pattern of the expected input data.

Do steps 2 and 3 on both client and server side-- A. for safety, and B. in case the client has scripts disabled, then C. for user conveinience and more professional looking results if scripts are allowed client-side.

To test just use console.log to output results in the debugger on client-side; server-side, same if node.js or use equivalent if other language.