Optimizing Data Collection for Machine Learning

10/03/2022
by   Rafid Mahmood, et al.
54

Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose a new paradigm for modeling the data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes to tasks requiring multiple data sources, such as labeled and unlabeled data used in semi-supervised learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which minimizes expected future collection costs. Finally, we numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.

READ FULL TEXT
research
05/05/2021

Using Synthetic Data to Enhance the Accuracy of Fingerprint-Based Localization: A Deep Learning Approach

Human-centered data collection is typically costly and implicates issues...
research
07/04/2022

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Given a small training data set and a learning algorithm, how much more ...
research
05/10/2021

Efficient Self-Supervised Data Collection for Offline Robot Learning

A practical approach to robot reinforcement learning is to first collect...
research
09/17/2020

CARONTE: Crawling Adversarial Resources Over Non-Trusted, High-Profile Environments

The monitoring of underground criminal activities is often automated to ...
research
09/28/2020

Reactive Supervision: A New Method for Collecting Sarcasm Data

Sarcasm detection is an important task in affective computing, requiring...
research
11/20/2017

Data Capture & Analysis to Assess Impact of Carbon Credit Schemes

Data enables Non-Governmental Organisations (NGOs) to quantify the impac...
research
02/17/2019

A semi-supervised deep residual network for mode detection in Wi-Fi signals

Due to their ubiquitous and pervasive nature, Wi-Fi networks have the po...

Please sign up or login with your details

Forgot password? Click here to reset