MDB: Interactively Querying Datasets and Models

08/13/2023

∙

As models are trained and deployed, developers need to be able to systematically debug errors that emerge in the machine learning pipeline. We present MDB, a debugging framework for interactively querying datasets and models. MDB integrates functional programming with relational algebra to build expressive queries over a database of datasets and model predictions. Queries are reusable and easily modified, enabling debuggers to rapidly iterate and refine queries to discover and characterize errors and model behaviors. We evaluate MDB on object detection, bias discovery, image classification, and data imputation tasks across self-driving videos, large language models, and medical records. Our experiments show that MDB enables up to 10x faster and 40% shorter queries than other baselines. In a user study, we find developers can successfully construct complex queries that describe errors of machine learning models.

READ FULL TEXT

MDB: Interactively Querying Datasets and Models

Text2Cohort: Democratizing the NCI Imaging Data Commons with Natural Language Cohort Discovery

Learning to Name Classes for Vision and Language Models

Data-Free Model Extraction Attacks in the Context of Object Detection

Explaining Wrong Queries Using Small Examples

Do Machine Learning Models Learn Common Sense?

VLSlice: Interactive Vision-and-Language Slice Discovery

On the Expressiveness of LARA: A Unified Language for Linear and Relational Algebra

MDB: Interactively Querying Datasets and Models

Related Research

Text2Cohort: Democratizing the NCI Imaging Data Commons with Natural Language Cohort Discovery

Learning to Name Classes for Vision and Language Models

Data-Free Model Extraction Attacks in the Context of Object Detection

Explaining Wrong Queries Using Small Examples

Do Machine Learning Models Learn Common Sense?

VLSlice: Interactive Vision-and-Language Slice Discovery

On the Expressiveness of LARA: A Unified Language for Linear and Relational Algebra