Has anyone ever tested a row based data de-duplication and matching program (the data could be any personal information like names,addresses and postal codes etc) ?

What kind of strategies/tools have been most effective to ascertain the quality of such programs and avoid regressions if its under active development ?

We would like to be able to automate the large part of our verification process and hence would like to know about automation tools/strategy applied in a similar context from someone who has done this kind of stuff already.

The software has as UI (java fx), an existing api which is exposed through web services as well. The quality concerns are around the matching logic and scalability as it needs to be able to process fairly large data in no time.

  • Is the web service a public API, or does it exist only to support the JavaFX UI? Also, what can you tell us about the matching criteria?
    – user246
    Commented Jul 5, 2011 at 19:54
  • @user246 .Yes the web service is an public API. The matching criteria is configurable and score based so non-exact matches can also feature as matches (partial though) depending on the scoring and respective weightage of the cols. being compared.
    – Rajneesh
    Commented Jul 18, 2011 at 7:39
  • Might be a silly idea but... have you looked at research articles how they evaluate algorithms for removing duplicates? scholar.google.pl/scholar?q=data+cleaning+duplicates
    – dzieciou
    Commented Oct 19, 2013 at 10:20

1 Answer 1


I can't promise that our techniques are state-of-the-art, but they gave us the level of confidence we needed and might give you some ideas for an approach.

We made a backup of the production database before we made the fix, with all of its known duplications. We then ran the de-duping code against that database and analyzed the duplications it found to ensure they were true dupes, and verified that it was finding all of the known actual dupes.

In addition, we created regression E2E test cases that would populate a clean database with data that covered as much of our de-duping logic as we had time to test, and then would query the database after the service was done running to verify that only the expected data remained. We already had code available to set up clean databases automatically and call the services, then run the queries and log the results; if we hadn't been able to leverage these tools, we might have used a different approach.

For performance and scalability, we timed the de-duping program running against the backup of our existing database in staging while forwarding current production traffic to the staging server to determine how many records per second were being de-duped while dealing with a standard load. We were mostly concerned with (a) if de-duping would affect the processing current traffic, and (b) how long the de-duping logic would take to catch up with current traffic. If we needed a reproducible performance test, I think we would have written a simulator program that would drop files on a pre-programmed schedule to simulate production traffic.

  • Some excellent points here. In particular, "populate a clean database with data that covered as much of our de-duping logic as we had time to test, and then would query the database after the service was done running to verify that only the expected data remained." is one of the keys to success, IMHO. Commented Jul 6, 2011 at 19:01
  • @Ethel.. I do see where you are coming from and its looks to me a nice solution but if we think of non-persistent systems and ability to do score based duplication (non-exact partial matches); things become quite complex.
    – Rajneesh
    Commented Jul 22, 2011 at 2:45
  • @Rajneesh, I believe it. I only answered at all because no one else had put an answer down yet (note the date of my answer is two days after your post). I was hoping people would say, "Geesh, what an idiot! We do things better than that!" and post alternatives :) Commented Jul 22, 2011 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.