5

My current assignment requires me to test data in a large number of text files ~ 200, usually ranging from 1GB to 20GB. The data is in the same schema with coordinates.

As of now, the approach is to load these files in a database (PostgreSQL) and then fire test cases using Python. Running a test on all the files (~80 GB) takes anywhere around 7/8 hours.

Even though the process is majorly automated, there are few drawbacks, namely a lack of proper reporting and manual intervention.

I want to optimise and improve this process.

Is the above approach is the best way of testing data? Are there better ways to do this?

What are the approaches you guys use to test data?

Can data languages like R / Scala be useful for this kind of testing?

Looking for suggestions and pointers.

Thanks.

Improve this question
13
  • 1
    what problem are you trying to solve ? you can solve performance issues by moving to C and having better hardware, or by using distributed computing and storage like Hadoop.
    – Rsf
    Sep 27, 2017 at 13:34
  • 8
    8 hours to process 80GB of data seems a long time. Did you identify the bottlenecks? (by profiling) Is it comparing parsed data against some "golden" pattern, and bottleneck is to access the "golden" value? Is bottleneck some calculation - can you make it faster? Changing to R will not make it faster if R will be doing same slow calls. Don't guess - measure.
    – Peter M. - stands for Monica
    Sep 27, 2017 at 14:11
  • 1
    Do you have baseline files? If so, you may be better on a simple comparison so that if the test file and the baseline file are the same, you don't need to inspect any closer.
    – Kate Paulk ♦
    Sep 27, 2017 at 14:49
  • 1
    What are you trying to do? Validate the data is correct/meets some format? Where is the data coming from? What are you validating it separately?
    – Kevin McKenzie
    Sep 27, 2017 at 17:34
  • 2
    I'm not completely sure this is the appropriate place for this question, as you're doing data validation, not software validation, and performance/best practices of data processing tools might be better asked in other places that cater more to data scientists, but that said . . . I think you need to decide what problem you're trying to solve. You say you aren't time constrained; are you interested doing better reporting/error recovery? Something like spark would make it easier to shard the data processing, I suspect.
    – Kevin McKenzie
    Oct 3, 2017 at 14:49

1 Answer 1

Reset to default
1

If you are using HDD: 10000 mb * 200 / 40 mb/s = 50000 seconds to read all files from single hdd which is around 15 hours. If you are using SSD, it is around 5 hours. You can speed up by using multiple SSDs. Or creating virtual HDDs in ram memory and writing and reading from there. But most real thing - claster your environment to several pc or several SSDs.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.