2

I am trying to find a text comparison tool which can compare text imported from a source and target database. the files are CSV's and could contain millions of records. This is to test a data migration process. Any suggestions are appreciated.

Beyond compare is considered now, not sure how it will behave with larger volume files. Also let me know if there is a better approach to test two offline databases.

3
  • Before I give an answer, I'd like a little more information: are you looking for a primarily programmatic solution? Are there any data transformations involved? Will you need to repeat the tests or do you only need to test the process once? Please edit the question to give some more information – Kate Paulk Feb 13 '19 at 12:15
  • I would prefer a tool/utility which can be just re used. This is just a migration for DB version upgrade, so no transformations or data type change needed. Migration test would be a one time task to make sure it is successful. – CuriousQE Feb 14 '19 at 15:16
  • RedGate's SQL Compare and SQL Data Compare would do the job, but you'd want to be really sure you could complete all your testing within the trial period, or able to purchase them, because the tools are not cheap. They're extremely useful, but not cheap. – Kate Paulk Feb 14 '19 at 18:56
1

CSVDiff may help you.

Example:

CSV A

id,name,amount
1,bob,20
2,eva,63
3,sarah,7
4,jeff,19
6,fred,10

CSV B

id,name,amount
1,bob,23 
3,sarah,7
4,jeff,19
5,mira,81
6,fred,13

Results:

csvdiff --style=summary id a.csv b.csv

1 rows removed (20.0%)
1 rows added (20.0%)
2 rows changed (40.0%)

And you can save the results in a file. Pretty printing is easy; additionally you can build another tool to inspect it and report:

$ csvdiff --style=pretty --output=diff.json id a.csv b.csv
$ cat diff.json
{
  "_index": [
    "id"
  ],
  "added": [
    {
      "amount": "81",
      "id": "5",
      "name": "mira"
    }
  ],
  "changed": [
    {
      "fields": {
        "amount": {
          "from": "20",
          "to": "23"
        }
      },
      "key": [
        "1"
      ]
    },
    {
      "fields": {
        "amount": {
          "from": "10",
          "to": "13"
        }
      },
      "key": [
        "6"
      ]
    }
  ],
  "removed": [
    {
      "amount": "63",
      "id": "2",
      "name": "eva"
    }
  ]
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.