Yes this is a useful strategy in converting from one system to another as a one time event (it might span a couple of iterations but the idea is very short term).
You could effectively view this as BDD with 'half (the tests) already done'. You have the tests against data that works. You create a new back-end version and see if it does the same.
As I ...
ETL Testing is one of the scenarios where the testing is straightforward but the coding is complicated. As an overview, you should be looking to test each of the elements: Extract, Transform and Load individually and then all of them again as an integrated process.
In the systems I've tested in the past, failures would be flagged and stored, allowing a ...
I would split your question into two parts:
What needs to be tested?
You have already covered some critical scenarios. Besides these following should also be covered-:
-Ensure that all expected data is loaded into target table.
-Compare record counts between source and target.
-Check for any rejected records
-Null, non-unique or out of range ...
Comparing the old system to the new system given identical inputs and expecting identical outputs is a legitimate test. There are problems, however:
a) When the legacy system is retired, the Source of Truth is no longer available and comparisons are impossible. You'll have to be very comfortable that the systems work similarly enough to satisfy business ...
How complicated is the ETL logic? Are you bringing over every record? Do you transform the data? I would create targeted test data to test the logic itself.
Once you know what you know exactly which kind of data is brought over and exactly how the data is transformed you can absolutely automate it. Start with an empty database, insert to source for your ...
Answer1: As soon as the development is completed, testing should take place in dedicated testing environment. Testing environment is very small in terms of data volume but is just like production environment otherwise (for example, name of databases, schema, name of jobs and scheduling should be same as that of production).
Smaller data set guarantees that ...
I think it will depend on the nature of the testing itself. Are you testing with large amount of production data? Are you testing with specially crafted test data?
To me if I am focused on ETL logic, the ETL is environment independent. It has a source and destination and thats pretty much it. If the test data is difficult to create and manage then that ...
Yes and No. The developers mapping should not be a secret, but what if the developer misinterpreted the business requirement and their mapping is wrong? I would first validate that the mapping matches the requirements (and that nothing was overlooked) and then use the mapping as a basis for generating test cases.
Given you're talking about "millions" it makes sense to use a load testing tool which can do multiple actions in parallel. If you don't have a better candidate yet you can consider Apache JMeter which provides:
CSV Data Set Config - so you can read "expected" data from file(s) in your local file system
JDBC Request sampler - to execute ...
Goal of whole organization (business, developers, QA, etc) is to deliver the best quality product to your customers for the best price.
If mapping contributes to this goal, it should be done. If contribution is not significant, it is waste of efforts.
Your QA manager need to talk to devs' manager and business manager how to accomplish that goal. Is the ...
The Source to Destination mappings is some kind of input for the Testing Phase but not exactly.
The Source data may be in the different format like DB, CSV, Excel, XML & JSON.
Each format file should have the header pieces of information like
Ex: Customer_ID is the header information.
The Source header pieces of information, we need to map to the ...
You may want to look for MapReduce techniques, in tools such as Hadoop.
MapReduce breaks sets of data for distributed computing (mapping) and summarizes (reducing) results in a distributed fashion as well, incrementally.
From the Impala website I read it supports ODBC and SQL, so you could use regular database tools to help with manual or automated testing.
Sounds like an ETL process. You have an extracted file from somewhere, you have an application that transforms and loads it into an Impala datastore.
That's a pain. The tools which claim to be able to do it require a fat wallet and expertise. This is basically a DB migration project.
A DIY way would be:
Connect to your Oracle DB with any tool and copy the table structures.
Find the SQL Server data type equivalent of each column
Create the same tables in SQL Server with the adequate types
Dump the data ...
Source to target mapping must be a part of inputs shared with the testing team
I have worked as an ETL tester for about 4 years for one of our US clients. It must be kept up to date at a common repository accessible to testers and developers.
1. Why testers need mapping document?
The reason is; because it is the encyclopedia for each and every table that ...
The best way to ensure the completeness and correctness of data is by testing it in SQL.
Can you try creating a database on your machine and use import\export wizard in SQL server and import the source and target file in two tables? I think then you will be able to validate the data transformation and ETL logic
I am also curious to know if there is any ...
Please find below the list of ETL testing scenarios for beginners:
Validation of Mapping Document
A mapping document serves as a requirement specification document for the ETL testers. Testers should verify that the mapping document has all the required information which is as follows:-
• Source Database information
• Transformation rules
• Target ...