Questions tagged [etl]
Extract, Transform, Load (ETL) refers to a process in database usage and especially in data warehousing that performs data extraction, data transformation and data loading.
16
questions
0
votes
1
answer
160
views
Validate Data completeness
Currently all database data is migrated from SQL/Oracle to snowflake( i.e. Extraction and Load only), basically a replica of Data so that later we can modify the data and Power Bi can fetch from ...
- 103
-1
votes
1
answer
92
views
What are the validations I need to perform before transferring data from source system to target system and if my source system doesn't any have data?
What are the validations I need to perform
Before transferring data from source system to target system?
If my source system doesn't have data to transfer to DWH, how do I test in that scenario?
-1
votes
1
answer
678
views
How to do ETL Automation Testing using C# or Python
In my present project I m testing on reports which needs to be checked everyday manually by executing SQL queries for all columns from source tables, Dims and Facts tables and destination tables. Is ...
- 11
1
vote
2
answers
991
views
How to check the data completeness in ETL Testing
I am working on healthcare project. I need to verify the data present in staging tables, DWH tables and target tables. I am checking manually for few records but my management asking to look for a ...
- 11
4
votes
3
answers
436
views
Can existing tests and data effectively compare old and new systems?
This idea was presented to me recently:
You have a complex legacy ETL system in Java in a TEST environment that no one understands completely. You write some tests that run it and compare the output ...
- 149
0
votes
1
answer
58
views
How can we use batch processing in ETL?
Can anyone tell me how to leverage batch processing in ETL? I have been working in ETL and testing for a year now but haven't used batch processing. Can anyone guide me to any nice read or a small ...
0
votes
1
answer
193
views
validating Large number of records
What are the primary testing techniques/methods to validate the every data point in the rows of records on source side to Target. How do we handle when there are millions of records to be validated ...
1
vote
1
answer
588
views
Data ingestion testing
In one of my project that I am working on, data source is a file and the data in file has to be copied in Impala (Hadoop).
Development work is already done and I need to figure out a testing strategy ...
- 185
0
votes
1
answer
75
views
What is the best approach/tools to test data copy from Oracle to SQL Server?
I have test data copy from Oracle database to SQL Server.
Data will be copied every hour. Data in Oracle is constantly changed and in SQL Server it will be in read-only mode.
- 101
3
votes
5
answers
932
views
ETL Quality Assurance - is source-to-target mapping consider an input to the testing phase?
At my place of work we have long debated whether source-to-target mapping should be an input to the testing phase perform by QA.
Developers insist that QAs should follow the requirements documents, ...
- 131
3
votes
2
answers
418
views
aws data pipeline testing tools
Our team is building an AWS product which uses several sets of tools like Lambda, SNS topics, EC2, data pipelines etc. I would like to know what type of testing tools/strategies that can be used to ...
2
votes
3
answers
18k
views
I have an ETL scenario where my source is Database and Target is flat file. How best can I test it to make bug free?
The flat file has millions of records and the source query is joining at least 10 tables.
The Testing I have done is:
Structure validation of the file. Like name & number of fields, delimiter, ...
- 21
3
votes
2
answers
861
views
Is testing environment for ETL testing required
My question is, where does the testing take place?
Do we need to set up separate testing environment for ETL testing?
Second, once the data is loaded in staging, testing starts or you wait till it ...
- 41
6
votes
4
answers
3k
views
Term for checking the correctness of data
We have a systems that pulls large sums of data from an external system then we transform the data and afterwards we need to verify this transformed data is correct.
How do we call this type of ...
- 32.5k
6
votes
3
answers
4k
views
ETL testing- checklist for testing
Can anyone tell me the basic test scenarios and checklist for ETL process testing for beginner level with any example ?
- 879
8
votes
5
answers
20k
views
What are some real world strategies for testing ETL and data warehouse applications
Does anyone have any advice to share on how to successfully test BIG ETL or data warehouse applications?
I have my own ideas of-course, but there only appears to be one main article on line that ...
- 9,750