Questions tagged [etl]

Extract, Transform, Load (ETL) refers to a process in database usage and especially in data warehousing that performs data extraction, data transformation and data loading.

Filter by
Sorted by
Tagged with
0 votes
1 answer

Validate Data completeness

Currently all database data is migrated from SQL/Oracle to snowflake( i.e. Extraction and Load only), basically a replica of Data so that later we can modify the data and Power Bi can fetch from ...
user166013's user avatar
-1 votes
1 answer

What are the validations I need to perform before transferring data from source system to target system and if my source system doesn't any have data?

What are the validations I need to perform Before transferring data from source system to target system? If my source system doesn't have data to transfer to DWH, how do I test in that scenario?
Umesh Borse's user avatar
-1 votes
1 answer

How to do ETL Automation Testing using C# or Python

In my present project I m testing on reports which needs to be checked everyday manually by executing SQL queries for all columns from source tables, Dims and Facts tables and destination tables. Is ...
user46586's user avatar
1 vote
2 answers

How to check the data completeness in ETL Testing

I am working on healthcare project. I need to verify the data present in staging tables, DWH tables and target tables. I am checking manually for few records but my management asking to look for a ...
user46586's user avatar
4 votes
3 answers

Can existing tests and data effectively compare old and new systems?

This idea was presented to me recently: You have a complex legacy ETL system in Java in a TEST environment that no one understands completely. You write some tests that run it and compare the output ...
xdhmoore's user avatar
  • 149
0 votes
1 answer

How can we use batch processing in ETL?

Can anyone tell me how to leverage batch processing in ETL? I have been working in ETL and testing for a year now but haven't used batch processing. Can anyone guide me to any nice read or a small ...
confusedQA's user avatar
0 votes
1 answer

validating Large number of records

What are the primary testing techniques/methods to validate the every data point in the rows of records on source side to Target. How do we handle when there are millions of records to be validated ...
user37635's user avatar
1 vote
1 answer

Data ingestion testing

In one of my project that I am working on, data source is a file and the data in file has to be copied in Impala (Hadoop). Development work is already done and I need to figure out a testing strategy ...
brij's user avatar
  • 185
0 votes
1 answer

What is the best approach/tools to test data copy from Oracle to SQL Server?

I have test data copy from Oracle database to SQL Server. Data will be copied every hour. Data in Oracle is constantly changed and in SQL Server it will be in read-only mode.
JaPyR's user avatar
  • 101
3 votes
5 answers

ETL Quality Assurance - is source-to-target mapping consider an input to the testing phase?

At my place of work we have long debated whether source-to-target mapping should be an input to the testing phase perform by QA. Developers insist that QAs should follow the requirements documents, ...
0x4B1D's user avatar
  • 131
3 votes
2 answers

aws data pipeline testing tools

Our team is building an AWS product which uses several sets of tools like Lambda, SNS topics, EC2, data pipelines etc. I would like to know what type of testing tools/strategies that can be used to ...
Lalindra Kawshika's user avatar
2 votes
3 answers

I have an ETL scenario where my source is Database and Target is flat file. How best can I test it to make bug free?

The flat file has millions of records and the source query is joining at least 10 tables. The Testing I have done is: Structure validation of the file. Like name & number of fields, delimiter, ...
A Ahmed's user avatar
  • 21
3 votes
2 answers

Is testing environment for ETL testing required

My question is, where does the testing take place? Do we need to set up separate testing environment for ETL testing? Second, once the data is loaded in staging, testing starts or you wait till it ...
Kelly's user avatar
  • 41
6 votes
4 answers

Term for checking the correctness of data

We have a systems that pulls large sums of data from an external system then we transform the data and afterwards we need to verify this transformed data is correct. How do we call this type of ...
Niels van Reijmersdal's user avatar
6 votes
3 answers

ETL testing- checklist for testing

Can anyone tell me the basic test scenarios and checklist for ETL process testing for beginner level with any example ?
Anand Natarajan's user avatar
8 votes
5 answers

What are some real world strategies for testing ETL and data warehouse applications

Does anyone have any advice to share on how to successfully test BIG ETL or data warehouse applications? I have my own ideas of-course, but there only appears to be one main article on line that ...
Bruce McLeod's user avatar
  • 9,750