DeepAI AI Chat
Log In Sign Up

A manifold learning perspective on representation learning: Learning decoder and representations without an encoder

by   Viktoria Schuster, et al.
Københavns Uni

Autoencoders are commonly used in representation learning. They consist of an encoder and a decoder, which provide a straightforward way to map n-dimensional data in input space to a lower m-dimensional representation space and back. The decoder itself defines an m-dimensional manifold in input space. Inspired by manifold learning, we show that the decoder can be trained on its own by learning the representations of the training samples along with the decoder weights using gradient descent. A sum-of-squares loss then corresponds to optimizing the manifold to have the smallest Euclidean distance to the training samples, and similarly for other loss functions. We derive expressions for the number of samples needed to specify the encoder and decoder and show that the decoder generally requires much less training samples to be well-specified compared to the encoder. We discuss training of autoencoders in this perspective and relate to previous work in the field that use noisy training examples and other types of regularization. On the natural image data sets MNIST and CIFAR10, we demonstrate that the decoder is much better suited to learn a low-dimensional representation, especially when trained on small data sets. Using simulated gene regulatory data, we further show that the decoder alone leads to better generalization and meaningful representations. Our approach of training the decoder alone facilitates representation learning even on small data sets and can lead to improved training of autoencoders. We hope that the simple analyses presented will also contribute to an improved conceptual understanding of representation learning.


page 3

page 8


Loss Landscapes of Regularized Linear Autoencoders

Autoencoders are a deep learning model for representation learning. When...

Learning Discriminative Features using Encoder-Decoder type Deep Neural Nets

As machine learning is applied to an increasing variety of complex probl...

LegoNet: A Fast and Exact Unlearning Architecture

Machine unlearning aims to erase the impact of specific training samples...

On denoising autoencoders trained to minimise binary cross-entropy

Denoising autoencoders (DAEs) are powerful deep learning models used for...

On the Size and Width of the Decoder of a Boolean Threshold Autoencoder

In this paper, we study the size and width of autoencoders consisting of...

Shaping representations through communication: community size effect in artificial learning systems

Motivated by theories of language and communication that explain why com...

Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning

The encoder-decoder models for unsupervised sentence representation lear...