FEDEX: An Explainability Framework for Data Exploration Steps

09/13/2022
by   Daniel Deutch, et al.
0

When exploring a new dataset, Data Scientists often apply analysis queries, look for insights in the resulting dataframe, and repeat to apply further queries. We propose in this paper a novel solution that assists data scientists in this laborious process. In a nutshell, our solution pinpoints the most interesting (sets of) rows in each obtained dataframe. Uniquely, our definition of interest is based on the contribution of each row to the interestingness of different columns of the entire dataframe, which, in turn, is defined using standard measures such as diversity and exceptionality. Intuitively, interesting rows are ones that explain why (some column of) the analysis query result is interesting as a whole. Rows are correlated in their contribution and so the interesting score for a set of rows may not be directly computed based on that of individual rows. We address the resulting computational challenge by restricting attention to semantically-related sets, based on multiple notions of semantic relatedness; these sets serve as more informative explanations. Our experimental study across multiple real-world datasets shows the usefulness of our system in various scenarios.

READ FULL TEXT

page 9

page 10

research
04/07/2021

Correlation Sketches for Approximate Join-Correlation Queries

The increasing availability of structured datasets, from Web tables and ...
research
10/23/2020

An analysis of the SIGMOD 2014 Programming Contest: Complex queries on the LDBC social network graph

This report contains an analysis of the queries defined in the SIGMOD 20...
research
02/10/2021

Explaining Inference Queries with Bayesian Optimization

Obtaining an explanation for an SQL query result can enrich the analysis...
research
01/30/2023

Explaining Dataset Changes for Semantic Data Versioning with Explain-Da-V (Technical Report)

In multi-user environments in which data science and analysis is collabo...
research
02/03/2020

Optimizing Query Predicates with Disjunctions for Column Stores

Since its inception, database research has given limited attention to op...
research
03/21/2019

Explain3D: Explaining Disagreements in Disjoint Datasets

Data plays an important role in applications, analytic processes, and ma...
research
06/13/2018

Towards Semantically Enhanced Data Understanding

In the field of machine learning, data understanding is the practice of ...

Please sign up or login with your details

Forgot password? Click here to reset