I am working on a project which has big data components in it. What would be testing artifacts (types of testing) of it which consists of data batch jobs, data ingestion framework, HDFS, tableau visualisation, email scheduler?
Ideally we should have testing at all the layers : 1. Data ingestion 2. Data processing 3. HDFS 4. Data visualisation But what types of testing should be dpne at all layers ?