In our project, related to Big Data.
On daily basis we receive 20 to 30 xml files and using hive/pig scripts the XML are loaded to database. The files are placed outside Hadoop environment. sometimes they millions of records
Currently we are manually validating XML to XSD XSD to Table Structure spot check few XML records with table data
The ask from business team is to get
1) Total count of records from XML files 2) Generate distinct records from a column 3) See wether we can automate XML to table data validation
Is there a way to do it using any automation tools.
Thank you Pratibha