0

I have a Java batch job application that reads input data from an Oracle database, does a lots of calculations, handles many edge cases, and outputs results back into different database tables.

The calculations should always return the same results on my subset samples regardless of the application code changes.

Therefore, my idea is to run the batch jobs automatically on each new application version and compare the result data with expected results stored in some file.

Resulting test dataset size is on average 1,000,000 rows in about 20 tables which have about 10 columns each. I would like to check the results on the test environment and compare them with expected data stored in CSV or some other format (ignoring IDs, timestamps).

What would be the best approach to automate checking correctness of this resulting data?

Is there a framework suitable for this task, or should I write something completely custom? I've looked into DbUnit and Database Rider but I am unsure whether they are the right choice for this job. They seem to focus more on data generation, which I already have, and data cleanup, which I don't really need afterwards.

Improve this question

2 Answers 2

Reset to default
0

Resulting test dataset size is on average 1 000 000 rows in about 20 tables which have about 10 columns each.

You are dealing with two different concerns:

  1. The calculations are performed by Interactors by manipulating Entities. For checking these calculations, you can use any unit testing libraries to exercise these objects and use your test data set. Nothing related to databases are done here.
  2. The mapping of Entities into the databases is done on the Entity Gateways. In this case, you can use database-specific tools, but you don't need to deal with your large dataset, since the point here is only the mapping, not calculations.

enter image description here

Improve this answer
1
  • Well, I guess you are probably right about me using the wrong approach here. In a way I was trying to use a shortcut - checking the data which I know is correct instead of checking the application logic. The main reasons which even led me to my solution were: a) there are just so many edge cases here that impact these calculations, and they occur because of the data, it's really hard to cover them all in unit tests b) programmers are not writing any unit tests and I was unable to change that for many years. Not really sure what to do here.
    – fing
    Commented Dec 5, 2019 at 16:45
0

You can use Talend to write a job that compares a DB to a file.

See this video for an example of what Talend can do. You can use the tJoin component to check for data matches.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.