Quantization-Based Regularization for Autoencoders

05/27/2019
by   Hanwei Wu, et al.
0

Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based regularizer in the bottleneck stage of autoencoder models to learn meaningful latent representations. We combine both perspectives of Vector Quantized-Variational AutoEncoders (VQ-VAE) and classical denoising regularization schemes of neural networks. We interpret quantizers as regularizers that constrain latent representations while fostering a similarity mapping at the encoder. Before quantization, we impose noise on the latent variables and use a Bayesian estimator to optimize the quantizer-based representation. The introduced bottleneck Bayesian estimator outputs the posterior mean of the centroids to the decoder, and thus, is performing soft quantization of the latent variables. We show that our proposed regularization method results in improved latent representations for both supervised learning and clustering downstream tasks when compared to autoencoders using other bottleneck structures.

READ FULL TEXT
research
07/12/2018

Learning Product Codebooks using Vector Quantized Autoencoders for Image Retrieval

The Vector Quantized-Variational Autoencoder (VQ-VAE) provides an unsupe...
research
03/20/2023

Training Invertible Neural Networks as Autoencoders

Autoencoders are able to learn useful data representations in an unsuper...
research
07/19/2018

Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer

Autoencoders provide a powerful framework for learning compressed repres...
research
02/19/2020

Hierarchical Quantized Autoencoders

Despite progress in training neural networks for lossy image compression...
research
06/25/2020

A New Modal Autoencoder for Functionally Independent Feature Extraction

Autoencoders have been widely used for dimensional reduction and feature...
research
11/25/2022

Homology-constrained vector quantization entropy regularizer

This paper describes an entropy regularization term for vector quantizat...
research
05/12/2020

Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders

The latent variables learned by VAEs have seen considerable interest as ...

Please sign up or login with your details

Forgot password? Click here to reset