Log In Sign Up

Optimizing Data Collection for Machine Learning

by   Rafid Mahmood, et al.

Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose a new paradigm for modeling the data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes to tasks requiring multiple data sources, such as labeled and unlabeled data used in semi-supervised learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which minimizes expected future collection costs. Finally, we numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.


Using Synthetic Data to Enhance the Accuracy of Fingerprint-Based Localization: A Deep Learning Approach

Human-centered data collection is typically costly and implicates issues...

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Given a small training data set and a learning algorithm, how much more ...

Efficient Self-Supervised Data Collection for Offline Robot Learning

A practical approach to robot reinforcement learning is to first collect...

CARONTE: Crawling Adversarial Resources Over Non-Trusted, High-Profile Environments

The monitoring of underground criminal activities is often automated to ...

Reactive Supervision: A New Method for Collecting Sarcasm Data

Sarcasm detection is an important task in affective computing, requiring...

A semi-supervised deep residual network for mode detection in Wi-Fi signals

Due to their ubiquitous and pervasive nature, Wi-Fi networks have the po...

Data Capture & Analysis to Assess Impact of Carbon Credit Schemes

Data enables Non-Governmental Organisations (NGOs) to quantify the impac...