Technical Report: Optimizing Human Involvement for Entity Matching and Consolidation

by   Ji Sun, et al.

An end-to-end data integration system requires human feedback in several phases, including collecting training data for entity matching, debugging the resulting clusters, confirming transformations applied on these clusters for data standardization, and finally, reducing each cluster to a single, canonical representation (or "golden record"). The traditional wisdom is to sequentially apply the human feedback, obtained by asking specific questions, within some budget in each phase. However, these questions are highly correlated; the answer to one can influence the outcome of any of the phases of the pipeline. Hence, interleaving them has the potential to offer significant benefits. In this paper, we propose a human-in-the-loop framework that interleaves different types of questions to optimize human involvement. We propose benefit models to measure the quality improvement from asking a question, and cost models to measure the human time it takes to answer a question. We develop a question scheduling framework that judiciously selects questions to maximize the accuracy of the final golden records. Experimental results on three real-world datasets show that our holistic method significantly improves the quality of golden records from 70 approaches.


page 2

page 6

page 17


Entity Consolidation: The Golden Record Problem

Four key processes in data integration are: data preparation (i.e., extr...

AutoBlock: A Hands-off Blocking Framework for Entity Matching

Entity matching seeks to identify data records over one or multiple data...

WebGPT: Browser-assisted question-answering with human feedback

We fine-tune GPT-3 to answer long-form questions using a text-based web-...

Knowledge Questions from Knowledge Graphs

We address the novel problem of automatically generating quiz-style know...

Exploring Question-Specific Rewards for Generating Deep Questions

Recent question generation (QG) approaches often utilize the sequence-to...

Machop: an End-to-End Generalized Entity Matching Framework

Real-world applications frequently seek to solve a general form of the E...

Optimal Clustering from Noisy Binary Feedback

We study the problem of recovering clusters from binary user feedback. I...