In our project, related to Big Data.
On daily basis we receive 20 to 30
XML files and using
hive/pig scripts the XML are loaded to database. The files are placed outside
Hadoop environment. Sometimes they millions of records
Currently we are manually validating:
1. XML to XSD
2. XSD to Table Structure
3. spot check few XML records with table data
The ask from business team is to get:
1. Total count of records from XML files
2. Generate distinct records from a column
3. See whether we can automate XML to table data validation
Is there a way to do it using any automation tools?