Need help with XML Testing. 2

Question

In our project, related to Big Data.

On daily basis we receive 20 to 30 XML files and using hive/pig scripts the XML are loaded to database. The files are placed outside Hadoop environment. Sometimes they millions of records

Currently we are manually validating:

 1. XML to XSD 
 2. XSD to Table Structure 
 3. spot check few XML records with table data

The ask from business team is to get:

1. Total count of records from XML files 
2. Generate distinct  records from a column 
3. See whether we can automate XML to table data validation 

Is there a way to do it using any automation tools?

Answer 1

1. You can obtain the total number of records from XML using xPath count function against each of the file with summing up the result
2. How you generate distinct set of entries from a column depends on what kind of database you are using
3. If you know the logic how your xml is to be mapped to your table structure you can implement automated validation. To do that you need: 1 - XML parser that would read objects from your XML (if you have XSD schema for your files that would simplify the things since there are many solutions for automatic parser builders from existing XSD files), 2 - database connectivity for your particular database vendor and your programming language. So you read structure from a file, read structure from database, apply your logic to check if they match or not.