In our project, related to Big Data.

On daily basis we receive 20 to 30 XML files and using hive/pig scripts the XML are loaded to database. The files are placed outside Hadoop environment. Sometimes they millions of records

Currently we are manually validating:

 1. XML to XSD 
 2. XSD to Table Structure 
 3. spot check few XML records with table data

The ask from business team is to get:

1. Total count of records from XML files 
2. Generate distinct  records from a column 
3. See whether we can automate XML to table data validation 

Is there a way to do it using any automation tools?

  • These 3 tasks could be done in many different tools, so the question should be more about what makes sense in your situation, e.g. what scripting/programming languages does the rest of your team know, so you're not the only one who does some automation and others can't make any improvements to it.
    – pavelsaman
    Commented Dec 15, 2019 at 20:45

1 Answer 1

  1. You can obtain the total number of records from XML using xPath count function against each of the file with summing up the result
  2. How you generate distinct set of entries from a column depends on what kind of database you are using
  3. If you know the logic how your xml is to be mapped to your table structure you can implement automated validation. To do that you need: 1 - XML parser that would read objects from your XML (if you have XSD schema for your files that would simplify the things since there are many solutions for automatic parser builders from existing XSD files), 2 - database connectivity for your particular database vendor and your programming language. So you read structure from a file, read structure from database, apply your logic to check if they match or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.