From neural PCA to deep unsupervised learning

11/28/2014
by   Harri Valpola, et al.
0

A network supporting deep unsupervised learning is presented. The network is an autoencoder with lateral shortcut connections from the encoder to decoder at each level of the hierarchy. The lateral shortcut connections allow the higher levels of the hierarchy to focus on abstract invariant features. While standard autoencoders are analogous to latent variable models with a single layer of stochastic variables, the proposed network is analogous to hierarchical latent variables models. Learning combines denoising autoencoder and denoising sources separation frameworks. Each layer of the network contributes to the cost function a term which measures the distance of the representations produced by the encoder and the decoder. Since training signals originate from all levels of the network, all layers can learn efficiently even in deep networks. The speedup offered by cost terms from higher levels of the hierarchy and the ability to learn invariant features are demonstrated in experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/22/2014

Denoising autoencoder with modulated lateral connections learns invariant representations of natural images

Suitable lateral connections between encoder and decoder are shown to al...
research
04/30/2015

Lateral Connections in Denoising Autoencoders Support Supervised Learning

We show how a deep denoising autoencoder with lateral connections can be...
research
07/22/2021

HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding

An autoencoder-based codec employs quantization to turn its bottleneck l...
research
10/12/2019

Residual Encoder-Decoder Network for Deep Subspace Clustering

Subspace clustering aims to cluster unlabeled data that lies in a union ...
research
02/01/2019

Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models

Variational autoencoders (VAEs) have received much attention recently as...
research
02/03/2016

Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition

We present a spike-based unsupervised regenerative learning scheme to tr...
research
11/15/2020

Predictive Coding, Variational Autoencoders, and Biological Connections

This paper identifies connections between predictive coding, from theore...

Please sign up or login with your details

Forgot password? Click here to reset