WDC Products: A Multi-Dimensional Entity Matching Benchmark

01/23/2023
by   Ralph Peeters, et al.
0

The difficulty of an entity matching task depends on a combination of multiple factors such as the amount of corner-case pairs, the fraction of entities in the test set that have not been seen during training, and the size of the development set. Current entity matching benchmarks usually represent single points in the space along such dimensions or they provide for the evaluation of matching methods along a single dimension, for instance the amount of training data. This paper presents WDC Products, an entity matching benchmark which provides for the systematic evaluation of matching systems along combinations of three dimensions while relying on real-word data. The three dimensions are (i) amount of corner-cases (ii) generalization to unseen entities, and (iii) development set size. Generalization to unseen entities is a dimension not covered by any of the existing benchmarks yet but is crucial for evaluating the robustness of entity matching systems. WDC Products is based on heterogeneous product data from thousands of e-shops which mark-up products offers using schema.org annotations. Instead of learning how to match entity pairs, entity matching can also be formulated as a multi-class classification task that requires the matcher to recognize individual entities. WDC Products is the first benchmark that provides a pair-wise and a multi-class formulation of the same tasks and thus allows to directly compare the two alternatives. We evaluate WDC Products using several state-of-the-art matching systems, including Ditto, HierGAT, and R-SupCon. The evaluation shows that all matching systems struggle with unseen entities to varying degrees. It also shows that some systems are more training data efficient than others.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/31/2019

EntEval: A Holistic Evaluation Benchmark for Entity Representations

Rich entity representations are useful for a wide class of problems invo...
research
10/07/2021

Cross-Language Learning for Entity Matching

Transformer-based matching methods have significantly moved the state-of...
research
04/21/2016

Row-less Universal Schema

Universal schema jointly embeds knowledge bases and textual patterns to ...
research
07/22/2021

Frost: Benchmarking and Exploring Data Matching Results

"Bad" data has a direct impact on 88 losing 12 representations of the sa...
research
01/27/2021

Alaska: A Flexible Benchmark for Data Integration Tasks

Data integration is a long-standing interest of the data management comm...
research
05/12/2022

Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction

Entity matching (EM) is the most critical step for entity resolution (ER...
research
12/01/2022

xEM: Explainable Entity Matching in Customer 360

Entity matching in Customer 360 is the task of determining if multiple r...

Please sign up or login with your details

Forgot password? Click here to reset