In our project, related to Big Data.
On daily basis we receive 20 to 30 XML
files and using hive/pig scripts
the XML are loaded to database. The files are placed outside Hadoop
environment. Sometimes they millions of records
Currently we are manually validating:
1. XML to XSD
2. XSD to Table Structure
3. spot check few XML records with table data
The ask from business team is to get:
1. Total count of records from XML files
2. Generate distinct records from a column
3. See whether we can automate XML to table data validation
Is there a way to do it using any automation tools?