I am working on a project which has big data components in it. What would be testing artifacts (types of testing) of it which consists of data batch jobs, data ingestion framework, HDFS, tableau visualisation, email scheduler?
Ideally we should have testing at all the layers : 1. Data ingestion 2. Data processing 3. HDFS 4. Data visualisation But what types of testing should be dpne at all layers ?
This will depend on a number of factors, including: time/budget, data quality policy of your organization, data quality of the source dataset, complexity of each layer, etc.
A general guideline is, as with any testing, to ask yourself "what could go wrong here?" at each step.
For example, you could do data integrity at 1 and 2. But if you had a low budget or you were quite sure that the data being ingested is pretty clean (according to your organization's data quality policy) and/or the data ingestion process is fairly simple, you could do data integrity in 2 only.
You might also want to test data consistency and data completeness. All these can be tested at all steps, it just depends on the constraints I mentioned in the first paragraph.
