Questions tagged [etl]

Extract, Transform, Load (ETL) refers to a process in database usage and especially in data warehousing that performs data extraction, data transformation and data loading.

Filter by
Sorted by
Tagged with
3
votes
3answers
68 views

Can existing tests and data effectively compare old and new systems?

This idea was presented to me recently: You have a complex legacy ETL system in Java in a TEST environment that no one understands completely. You write some tests that run it and compare the output ...
0
votes
1answer
29 views

How can we use batch processing in ETL?

Can anyone tell me how to leverage batch processing in ETL? I have been working in ETL and testing for a year now but haven't used batch processing. Can anyone guide me to any nice read or a small ...
0
votes
1answer
49 views

validating Large number of records

What are the primary testing techniques/methods to validate the every data point in the rows of records on source side to Target. How do we handle when there are millions of records to be validated ...
1
vote
1answer
99 views

Data ingestion testing

In one of my project that I am working on, data source is a file and the data in file has to be copied in Impala (Hadoop). Development work is already done and I need to figure out a testing strategy ...
0
votes
1answer
34 views

What is the best approach/tools to test data copy from Oracle to SQL Server?

I have test data copy from Oracle database to SQL Server. Data will be copied every hour. Data in Oracle is constantly changed and in SQL Server it will be in read-only mode.
3
votes
5answers
576 views

ETL Quality Assurance - is source-to-target mapping consider an input to the testing phase?

At my place of work we have long debated whether source-to-target mapping should be an input to the testing phase perform by QA. Developers insist that QAs should follow the requirements documents, ...
2
votes
0answers
277 views

aws data pipeline testing tools

Our team is building an aws product which uses several set of tools like lambda, SNS topics,EC2, data pipelines etc. I would like to know what type of testing tools/strategies that can be used to ...
2
votes
3answers
7k views

I have an ETL scenario where my source is Database and Target is flat file. How best can I test it to make bug free?

The flat file has millions of records and the source query is joining at least 10 tables. The Testing I have done is: Structure validation of the file. Like name & number of fields, delimiter, ...
2
votes
2answers
394 views

Is testing environment for ETL testing required

My question is, where does the testing take place? Do we need to set up separate testing environment for ETL testing? Second, once the data is loaded in staging, testing starts or you wait till it ...
6
votes
4answers
2k views

Term for checking the correctness of data

We have a systems that pulls large sums of data from an external system then we transform the data and afterwards we need to verify this transformed data is correct. How do we call this type of ...
5
votes
3answers
2k views

ETL testing- checklist for testing

Can anyone tell me the basic test scenarios and checklist for ETL process testing for beginner level with any example ?
8
votes
4answers
19k views

What are some real world strategies for testing ETL and data warehouse applications

Does anyone have any advice to share on how to successfully test BIG ETL or data warehouse applications? I have my own ideas of-course, but there only appears to be one main article on line that ...