Analyzing noise in autoencoders and deep networks

06/06/2014
by   Ben Poole, et al.
0

Autoencoders have emerged as a useful framework for unsupervised learning of internal representations, and a wide variety of apparently conceptually disparate regularization techniques have been proposed to generate useful features. Here we extend existing denoising autoencoders to additionally inject noise before the nonlinearity, and at the hidden unit activations. We show that a wide variety of previous methods, including denoising, contractive, and sparse autoencoders, as well as dropout can be interpreted using this framework. This noise injection framework reaps practical benefits by providing a unified strategy to develop new internal representations by designing the nature of the injected noise. We show that noisy autoencoders outperform denoising autoencoders at the very task of denoising, and are competitive with other single-layer techniques on MNIST, and CIFAR-10. We also show that types of noise other than dropout improve performance in a deep network through sparsifying, decorrelating, and spreading information across representations.

READ FULL TEXT

page 6

page 8

research
09/09/2014

Winner-Take-All Autoencoders

In this paper, we propose a winner-take-all method for learning hierarch...
research
06/14/2018

Learning Dynamics of Linear Denoising Autoencoders

Denoising autoencoders (DAEs) have proven useful for unsupervised repres...
research
07/07/2019

Stacked autoencoders based machine learning for noise reduction and signal reconstruction in geophysical data

Autoencoders are neural network formulations where the input and output ...
research
01/26/2021

Blind Image Denoising and Inpainting Using Robust Hadamard Autoencoders

In this paper, we demonstrate how deep autoencoders can be generalized t...
research
02/12/2018

Recovering Loss to Followup Information Using Denoising Autoencoders

Loss to followup is a significant issue in healthcare and has serious co...
research
06/10/2020

DivNoising: Diversity Denoising with Fully Convolutional Variational Autoencoders

Deep Learning based methods have emerged as the indisputable leaders for...
research
08/29/2018

Autoencoders, Kernels, and Multilayer Perceptrons for Electron Micrograph Restoration and Compression

We present 14 autoencoders, 15 kernels and 14 multilayer perceptrons for...

Please sign up or login with your details

Forgot password? Click here to reset