DeepAI AI Chat
Log In Sign Up

Generalizing Dataset Distillation via Deep Generative Prior

by   George Cazenavette, et al.

Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. Despite recent progress in the field, existing dataset distillation methods fail to generalize to new architectures and scale to high-resolution datasets. To overcome the above issues, we propose to use the learned prior from pre-trained deep generative models to synthesize the distilled data. To achieve this, we present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space. Our method augments existing techniques, significantly improving cross-architecture generalization in all settings.


page 1

page 2

page 4

page 6

page 7

page 8


Dataset Distillation

Model distillation aims to distill the knowledge of a complex model into...

DiM: Distilling Dataset into Generative Model

Dataset distillation reduces the network training cost by synthesizing s...

Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models

Knowledge distillation in neural networks refers to compressing a large ...

Flexible Dataset Distillation: Learn Labels Instead of Images

We study the problem of dataset distillation - creating a small set of s...

NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistency

We introduce NaturalInversion, a novel model inversion-based method to s...

Boundary Optimizing Network (BON)

Despite all the success that deep neural networks have seen in classifyi...

Physics-aware Deep Generative Models for Creating Synthetic Microstructures

A key problem in computational material science deals with understanding...