Dataset Distillation

11/27/2018
by   Tongzhou Wang, et al.
3

Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few steps of gradient descent, given a particular fixed network initialization. We evaluate our method in a wide range of initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods in most settings.

READ FULL TEXT

page 2

page 7

research
05/02/2023

Generalizing Dataset Distillation via Deep Generative Prior

Dataset Distillation aims to distill an entire dataset's knowledge into ...
research
04/17/2021

Data Distillation for Text Classification

Deep learning techniques have achieved great success in many fields, whi...
research
10/06/2019

Improving Dataset Distillation

Dataset distillation is a method for reducing dataset sizes: the goal is...
research
11/18/2020

Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation

Knowledge Distillation is an effective method to transfer the learning a...
research
06/15/2020

Flexible Dataset Distillation: Learn Labels Instead of Images

We study the problem of dataset distillation - creating a small set of s...
research
07/16/2023

Dataset Distillation Meets Provable Subset Selection

Deep learning has grown tremendously over recent years, yielding state-o...
research
04/09/2018

Large scale distributed neural network training through online distillation

Techniques such as ensembling and distillation promise model quality imp...

Please sign up or login with your details

Forgot password? Click here to reset