0

In our project, related to Big Data.

On daily basis we receive 20 to 30 xml files and using hive/pig scripts the XML are loaded to database. The files are placed outside Hadoop environment. sometimes they millions of records

Currently we are manually validating XML to XSD XSD to Table Structure spot check few XML records with table data

The ask from business team is to get

              1) Total count of records from XML files 
              2) Generate distinct  records from a column 
              3) See wether we can automate XML to table data validation 

Is there a way to do it using any automation tools.

Thank you Pratibha

0
  1. You can obtain the total number of records from XML using xPath count function against each of the file with summing up the result
  2. How you generate distinct set of entries from a column depends on what kind of database you are using
  3. If you know the logic how your xml is to be mapped to your table structure you can implement automated validation. To do that you need: 1 - XML parser that would read objects from your XML (if you have XSD schema for your files that would simplify the things since there are many solutions for automatic parser builders from existing XSD files), 2 - database connectivity for your particular database vendor and your programming language. So you read structure from a file, read structure from database, apply your logic to check if they match or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.