On Memorization in Probabilistic Deep Generative Models

06/06/2021
by   Gerrit J. J. van den Burg, et al.
7

Recent advances in deep generative models have led to impressive results in a variety of application domains. Motivated by the possibility that deep learning models might memorize part of the input data, there have been increased efforts to understand how memorization can occur. In this work, we extend a recently proposed measure of memorization for supervised learning (Feldman, 2019) to the unsupervised density estimation problem and simplify the accompanying estimator. Next, we present an exploratory study that demonstrates how memorization can arise in probabilistic deep generative models, such as variational autoencoders. This reveals that the form of memorization to which these models are susceptible differs fundamentally from mode collapse and overfitting. Finally, we discuss several strategies that can be used to limit memorization in practice.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 6

page 15

page 16

02/17/2016

Auxiliary Deep Generative Models

Deep generative models parameterized by neural networks have recently ac...
11/08/2018

Bias and Generalization in Deep Generative Models: An Empirical Study

In high dimensional settings, density estimation algorithms rely crucial...
04/30/2019

Performing Structured Improvisations with pre-trained Deep Learning Models

The quality of outputs produced by deep generative models for music have...
09/07/2018

A simple probabilistic deep generative model for learning generalizable disentangled representations from grouped data

The disentangling problem is to discover multiple complex factors of var...
03/16/2016

One-Shot Generalization in Deep Generative Models

Humans have an impressive ability to reason about new concepts and exper...
07/16/2021

A comparative study of stochastic and deep generative models for multisite precipitation synthesis

Future climate change scenarios are usually hypothesized using simulatio...
06/15/2021

Divergence Frontiers for Generative Models: Sample Complexity, Quantization Level, and Frontier Integral

The spectacular success of deep generative models calls for quantitative...

Code Repositories

memorization

Code for "On Memorization in Probabilistic Deep Generative Models"


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.