3

My current assignment requires me to test data in a large number of text files ~ 200, usually ranging between 1GB to 20GB. The data is in same schema with coordinates.

As of now the approach is to load these files in a database (PostgreSQL) and then fire test cases using Python. Running test on all the files (~80 GB) takes anywhere around 7/8 hours.

Even though the process is majorly automated, there are few drawbacks, namely a lack of proper reporting and manual intervention.

I want to optimise and improve this process.

Is the above approach is the best way of testing data? Are there better ways to do this?

What are the approaches you guys use to test data?

Can data languages like R / Scala be useful for this kind of testing?

Looking for suggestions and pointers.

Thanks.

  • 1
    what problem are you trying to solve ? you can solve performance issues by moving to C and having better hardware, or by using distributed computing and storage like Hadoop. – Rsf Sep 27 '17 at 13:34
  • 7
    8 hours to process 80GB of data seems a long time. Did you identify the bottlenecks? (by profiling) Is it comparing parsed data against some "golden" pattern, and bottleneck is to access the "golden" value? Is bottleneck some calculation - can you make it faster? Changing to R will not make it faster if R will be doing same slow calls. Don't guess - measure. – Peter M. Sep 27 '17 at 14:11
  • 1
    Do you have baseline files? If so, you may be better on a simple comparison so that if the test file and the baseline file are the same, you don't need to inspect any closer. – Kate Paulk Sep 27 '17 at 14:49
  • 1
    What are you trying to do? Validate the data is correct/meets some format? Where is the data coming from? What are you validating it separately? – Kevin McKenzie Sep 27 '17 at 17:34
  • 2
    I'm not completely sure this is the appropriate place for this question, as you're doing data validation, not software validation, and performance/best practices of data processing tools might be better asked in other places that cater more to data scientists, but that said . . . I think you need to decide what problem you're trying to solve. You say you aren't time constrained; are you interested doing better reporting/error recovery? Something like spark would make it easier to shard the data processing, I suspect. – Kevin McKenzie Oct 3 '17 at 14:49
0

If you are using HDD: 10000 mb * 200 / 40 mb/s = 50000 seconds to read all files from single hdd which is around 15 hours. If you are using SSD, it is around 5 hours. You can speed up by using multiple SSDs. Or creating virtual HDDs in ram memory and writing and reading from there. But most real thing - claster your environment to several pc or several SSDs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.