Testing migrated data vs source data

Question

I am trying to find a text comparison tool which can compare text imported from a source and target database. the files are CSV's and could contain millions of records. This is to test a data migration process. Any suggestions are appreciated.

Beyond compare is considered now, not sure how it will behave with larger volume files. Also let me know if there is a better approach to test two offline databases.

Answer 1

CSVDiff may help you.

Example:

CSV A

id,name,amount
1,bob,20
2,eva,63
3,sarah,7
4,jeff,19
6,fred,10

CSV B

id,name,amount
1,bob,23 
3,sarah,7
4,jeff,19
5,mira,81
6,fred,13

Results:

csvdiff --style=summary id a.csv b.csv

1 rows removed (20.0%)
1 rows added (20.0%)
2 rows changed (40.0%)

And you can save the results in a file. Pretty printing is easy; additionally you can build another tool to inspect it and report:

$ csvdiff --style=pretty --output=diff.json id a.csv b.csv
$ cat diff.json
{
  "_index": [
    "id"
  ],
  "added": [
    {
      "amount": "81",
      "id": "5",
      "name": "mira"
    }
  ],
  "changed": [
    {
      "fields": {
        "amount": {
          "from": "20",
          "to": "23"
        }
      },
      "key": [
        "1"
      ]
    },
    {
      "fields": {
        "amount": {
          "from": "10",
          "to": "13"
        }
      },
      "key": [
        "6"
      ]
    }
  ],
  "removed": [
    {
      "amount": "63",
      "id": "2",
      "name": "eva"
    }
  ]
}