Group/cluster size decreases as data increases although same similar records number remains about the same

[rodrigo748]

Hello,

In a dataset of 2.6M records, that contains a set of 28 similar records, Zingg clustered these 28 similar records into 6 different z_cluster IDs.
Using the same model with a larger dataset of 14.5M records, that contains the exact same set of 28 similar records (plus 4, 32 in total), Zingg clustered them into 24 different z_cluster IDs.
Why would zingg do less matches when the number of total records increased (from 2.6M to 14.5M) but the set of same similar records was pretty much the same (28 vs 32)?

Thank you,
Rodrigo Escamilla

[sonalgoyal]

That’s very strange, never seen this. Do you have a reproducible test case we can use for debugging?

[rodrigo748]

I will schedule a meeting to show the issue because I am not sure how I can help reproduce this without the 14M records dataset.
I performed another test to confirm the issue:
First I created a very small dataset for training with combination or similar records which I know match and labeled them as matches. After training the model, I ran trainMatch phase and confirmed all records I labeled as matching had same z_cluster ID. The I ran the same model against the 14M dataset that also contains the matched records but this time zingg assigned different z_cluster IDs to those previously matched and properly correlated records. So same issue as in previous tests.
After training the model and saving the labels I change and rerun the query that populates the table which my inputPipe uses: inputPipe.addProperty("path","dbfs:/[path to hive table with 14M records]"). Then rerun trainMatch phase.

Thank you,
Rodrigo Escamilla

[sonalgoyal]

If you train again on the much larger dataset, the blocking model will be different. That is why it is recommended to train Zingg on the same data volumes as you prefer to run match on.

[rodrigo748]

Thank you for the response Sonal, but I have tried that following our first conversation. Trained on 32M records and generated 302 Matches, 890 Non-Matches. Results were very very fragmented. After that I tried this:
"Trained and matched on 2.6M records, that contain a set of 28 similar records, Zingg clustered these 28 similar records into 6 different z_cluster IDs. Using the same model with a larger dataset of 14.5M records, that contains the exact same set of 28 similar records (plus 4, 32 in total), Zingg clustered them into 24 different z_cluster IDs." and you requested a reproduceable test case for debugging this issue. That is why I scheduled the call to provide you what you need to reproduce this issue. Please let me know if you'd rather close this thread as unresolved.

Thank you,
Rodrigo Escamilla