DeepAI AI Chat
Log In Sign Up

Revisiting the probabilistic method of record linkage

by   Abel Dasylva, et al.

In theory, the probabilistic linkage method provides two distinct advantages over non-probabilistic methods, including minimal rates of linkage error and accurate measures of these rates for data users. However, implementations can fall short of these expectations either because the conditional independence assumption is made, or because a model with interactions is used but lacks the identification property. In official statistics, this is currently the main challenge to the automated production and use of linked data. To address this challenge, a new methodology is described for proper linkage problems, where matched records may be identified with a probability that is bounded away from zero, regardless of the population size. It models the number of neighbours of a given record, i.e. the number of resembling records. To be specific, the proposed model is a finite mixture where each component is the sum of a Bernoulli variable with an independent Poisson variable. It has the identification property and yields solutions for many longstanding problems, including the evaluation of blocking criteria and the estimation of linkage errors for probabilistic or non-probabilistic linkages, all without clerical reviews or conditional independence assumptions. Thus it also enables unsupervised machine learning solutions for record linkage problems.


page 1

page 2

page 3

page 4


Tractability through Exchangeability: A New Perspective on Efficient Probabilistic Inference

Exchangeability is a central notion in statistics and probability theory...

Breaking Bivariate Records

We establish a fundamental property of bivariate Pareto records for inde...

Supervised Negative Binomial Classifier for Probabilistic Record Linkage

Motivated by the need of the linking records across various databases, w...

Computational Phenotype Discovery via Probabilistic Independence

Computational Phenotype Discovery research has taken various pragmatic a...

Pragmatic Primitives for Non-blocking Data Structures

We define a new set of primitive operations that greatly simplify the im...

Electre Tri-Machine Learning Approach to the Record Linkage Problem

In this short paper, the Electre Tri-Machine Learning Method, generally ...

Linkage Free Dual System Estimation

In this paper it is shown that under certain conditions there is a relat...