DeepAI AI Chat
Log In Sign Up

Fundamentals of Task-Agnostic Data Valuation

by   Mohammad Mohammadi Amiri, et al.

We study valuing the data of a data owner/seller for a data seeker/buyer. Data valuation is often carried out for a specific task assuming a particular utility metric, such as test accuracy on a validation set, that may not exist in practice. In this work, we focus on task-agnostic data valuation without any validation requirements. The data buyer has access to a limited amount of data (which could be publicly available) and seeks more data samples from a data seller. We formulate the problem as estimating the differences in the statistical properties of the data at the seller with respect to the baseline data available at the buyer. We capture these statistical differences through second moment by measuring diversity and relevance of the seller's data for the buyer; we estimate these measures through queries to the seller without requesting raw data. We design the queries with the proposed approach so that the seller is blind to the buyer's raw data and has no knowledge to fabricate responses to queries to obtain a desired outcome of the diversity and relevance trade-off.We will show through extensive experiments on real tabular and image datasets that the proposed estimates capture the diversity and relevance of the seller's data for the buyer.


page 1

page 2

page 3

page 4


Evaluation of Genotypic Diversity Measurements Exploited in Real-Coded Representation

Numerous genotypic diversity measures (GDMs) are available in the litera...

Simultaneous Relevance and Diversity: A New Recommendation Inference Approach

Relevance and diversity are both important to the success of recommender...

Probabilistic Time Series Forecasting with Structured Shape and Temporal Diversity

Probabilistic forecasting consists in predicting a distribution of possi...

Efficient SPARQL Autocompletion via SPARQL

We show how to achieve fast autocompletion for SPARQL queries on very la...

ISP-Agnostic Image Reconstruction for Under-Display Cameras

Under-display cameras have been proposed in recent years as a way to red...

Transferring Knowledge from Text to Predict Disease Onset

In many domains such as medicine, training data is in short supply. In s...

Agnostic learning with unknown utilities

Traditional learning approaches for classification implicitly assume tha...