DeepAI AI Chat
Log In Sign Up

Labels, Information, and Computation: Efficient, Privacy-Preserving Learning Using Sufficient Labels

by   Shiyu Duan, et al.

In supervised learning, obtaining a large set of fully-labeled training data is expensive. We show that we do not always need full label information on every single training example to train a competent classifier. Specifically, inspired by the principle of sufficiency in statistics, we present a statistic (a summary) of the fully-labeled training set that captures almost all the relevant information for classification but at the same time is easier to obtain directly. We call this statistic "sufficiently-labeled data" and prove its sufficiency and efficiency for finding the optimal hidden representations, on which competent classifier heads can be trained using as few as a single randomly-chosen fully-labeled example per class. Sufficiently-labeled data can be obtained from annotators directly without collecting the fully-labeled data first. And we prove that it is easier to directly obtain sufficiently-labeled data than obtaining fully-labeled data. Furthermore, sufficiently-labeled data naturally preserves user privacy by storing relative, instead of absolute, information. Extensive experimental results are provided to support our theory.


page 1

page 2

page 3

page 4


Labeled Data Generation with Inexact Supervision

The recent advanced deep learning techniques have shown the promising re...

An analysis of over-sampling labeled data in semi-supervised learning with FixMatch

Most semi-supervised learning methods over-sample labeled data when cons...

Data Distillation: Towards Omni-Supervised Learning

We investigate omni-supervised learning, a special regime of semi-superv...

Semi-supervised learning by selective training with pseudo labels via confidence estimation

We propose a novel semi-supervised learning (SSL) method that adopts sel...

Renal Cell Carcinoma Detection and Subtyping with Minimal Point-Based Annotation in Whole-Slide Images

Obtaining a large amount of labeled data in medical imaging is laborious...

Multi-Class 3D Object Detection with Single-Class Supervision

While multi-class 3D detectors are needed in many robotics applications,...

Learning Concept Abstractness Using Weak Supervision

We introduce a weakly supervised approach for inferring the property of ...