Clustering method for 1 to n relationships and exporting predictions

[tkeyo]

Hi,

I am new to entity resolution and I found your library and examples super useful.

I have 2 questions though:

1. How to get 1-to-many matches?
   For context: I am trying to match multiple products in dataset A to 1 product in dataset B. Where dataset A has 2000 products and dataset B has 500 products.
   q1: Should I stop at the entity matching step and try to export the pairs from this step?
   q2: What would be the best clustering method for this requirement?

2. Exporting the results
   How can I export the predicted results for each of the products in dataset B i.e. get multiple products from dataset B matched to 1 product in dataset A.

Thank you.

[Nikoletos-K]

Hi!

Apologies for the late reply; we were off due to the summer break.

1. We don't support this exact feature because our clustering methods typically produce either one-to-one pairs or sets of candidate pairs in cases of Dirty ER. However, to achieve something similar to what you're looking for, you could consider taking the blocks after FAISS using this method. In this approach, the blocks contain the top-k most similar entities from D1 for every D2 entity, resulting in a 1 D2 entity to top-k D1 entities mapping. For more details on how to implement this workflow, you can refer to this tutorial.

2. You can use this method after executing the 'build_blocks' method, with the 'prediction' argument set to the blocks.

If you have any further questions, please don't hesitate to reach out!

Cheers,
Konstantinos

[tkeyo]

Hi @Nikoletos-K , no worries. I appreciate your reply nonetheless.

1. Candidate pairs

We don't support this exact feature because our clustering methods typically produce either one-to-one pairs or sets of candidate pairs in cases of Dirty ER.

So if I understand correctly - with Clustering methods like CenterClustering it's possible to achieve an output where one product in dataset B has multiple candidates from dataset A ? And in case I wanted 1-to-1 I could use something like UniqueMappingClustering?

However, to achieve something similar to what you're looking for, you could consider taking the blocks after FAISS using this method. In this approach, the blocks contain the top-k most similar entities from D1 for every D2 entity, resulting in a 1 D2 entity to top-k D1 entities mapping.
Thanks will take a deeper look - I was running into issues when trying to use VectorBasedBlocking.

I experimented with BLAST to achieve this - Generate blocks of similar products. Now, if I understand correctly BLAST will be building blocks of similar products within the full dataset regardless of it's source? So it can happen that a block will contain products from dataset A ?

Example if expected output

B	A
product B1	product A101
product B1	product A225
product B1	product A301
...	...
product B2	product A302
product B2	product A301

2. Thanks, after digging into the code I found this method 👍.

[Nikoletos-K]

Yeah, Unique Mapping is the best for 1-1 matches, and yes by using CenterClustering this could happen. BLAST is a comparison cleaning method among others.

I strongly suggest you to test and experiment with the nearest neighbor workflow, as we have seen that yields really good results.