DeepAI AI Chat
Log In Sign Up

A Formal Framework For Probabilistic Unclean Databases

by   Christopher De Sa, et al.

Traditional modeling of inconsistency in database theory casts all possible "repairs" equally likely. Yet, effective data cleaning needs to incorporate statistical reasoning. For example, yearly salary of 100k and age of 22 are more likely than 100k and 122 and two people with same address are likely to share their last name (i.e., a functional dependency tends to hold but may occasionally be violated). We propose a formal framework for unclean databases, where two types of statistical knowledge are incorporated. The first represents a belief of how intended (clean) data is generated, and the second represents a belief of how the actual database is realized through the introduction of noise. Formally, a Probabilistic Unclean Database (PUD) is a triple that consists of a probabilistic database that we call the "intention", a probabilistic data transformator that we call the "realization", and a dirty database that we call the "observation". We define three computational problems in this framework: cleaning (find the most likely intention), probabilistic query answering (compute the probability of an answer tuple), and learning (find the most likely parameters given examples of clean and dirty databases). We illustrate the framework on concrete representations of PUDs, show that they generalize traditional concepts of repairs such as cardinality and value repairs, draw connection to consistent query answering, and prove tractability results. We further show that parameters can be learned in practical instantiations, and in fact, prove that under certain conditions we can learn directly from a single dirty database without any need for clean examples.


page 1

page 2

page 3

page 4


Intention as Commitment toward Time

In this paper we address the interplay among intention, time, and belief...

An epistemic approach to model uncertainty in data-graphs

Graph databases are becoming widely successful as data models that allow...

Distributed Transition Systems with Tags for Privacy Analysis

We present a logical framework that formally models how a given private ...

Believe It or Not: Adding Belief Annotations to Databases

We propose a database model that allows users to annotate data with beli...

A probabilistic database approach to autoencoder-based data cleaning

Data quality problems are a large threat in data science. In this paper,...

A New Rational Algorithm for View Updating in Relational Databases

The dynamics of belief and knowledge is one of the major components of a...

A Signaling Game Approach to Databases Querying and Interaction

As most database users cannot precisely express their information needs,...