Learning deep representations by mutual information estimation and maximization

08/20/2018
by   R Devon Hjelm, et al.
0

Many popular representation-learning algorithms use training objectives defined on the observed data space, which we call pixel-level. This may be detrimental when only a small fraction of the bits of signal actually matter at a semantic level. We hypothesize that representations should be learned and evaluated more directly in terms of their information content and statistical or structural constraints. To address the first quality, we consider learning unsupervised representations by maximizing mutual information between part or all of the input and a high-level feature vector. To address the second, we control characteristics of the representation by matching to a prior adversarially. Our method, which we call Deep INFOMAX (DIM), can be used to learn representations with desired characteristics and which empirically outperform a number of popular unsupervised learning methods on classification tasks. DIM opens new avenues for unsupervised learn-ing of representations and is an important step towards flexible formulations of representation learning objectives catered towards specific end-goals.

READ FULL TEXT
research
02/04/2020

Graph Representation Learning via Graphical Mutual Information Maximization

The richness in the content of various information networks such as soci...
research
02/20/2020

Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning

We introduce a parameterization method called Neural Bayes which allows ...
research
12/21/2019

Learning Representations by Maximizing Mutual Information in Variational Autoencoder

Variational autoencoders (VAE) have ushered in a new era of unsupervised...
research
06/14/2021

Which Mutual-Information Representation Learning Objectives are Sufficient for Control?

Mutual information maximization provides an appealing formalism for lear...
research
12/04/2021

Representation Learning for Conversational Data using Discourse Mutual Information Maximization

Although many pretrained models exist for text or images, there have bee...
research
08/05/2021

M2IOSR: Maximal Mutual Information Open Set Recognition

In this work, we aim to address the challenging task of open set recogni...
research
11/08/2018

On the Statistical and Information-theoretic Characteristics of Deep Network Representations

It has been common to argue or imply that a regularizer can be used to a...

Please sign up or login with your details

Forgot password? Click here to reset