Multimodal Representation Learning via Maximization of Local Mutual Information

03/08/2021
by   Ruizhi Liao, et al.
0

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method learns image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that, typically, the sum of local mutual information is a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/09/2020

Matching Text with Deep Mutual Information Estimation

Text matching is a core natural language processing research problem. Ho...
research
05/03/2020

Mutual Information Gradient Estimation for Representation Learning

Mutual Information (MI) plays an important role in representation learni...
research
03/28/2019

Wasserstein Dependency Measure for Representation Learning

Mutual information maximization has emerged as a powerful learning objec...
research
12/09/2019

Learning Disentangled Representations via Mutual Information Estimation

In this paper, we investigate the problem of learning disentangled repre...
research
05/14/2021

Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representations

Recently, maximizing mutual information has emerged as a powerful method...
research
10/07/2021

InfoSeg: Unsupervised Semantic Image Segmentation with Mutual Information Maximization

We propose a novel method for unsupervised semantic image segmentation b...
research
03/13/2020

DHOG: Deep Hierarchical Object Grouping

Recently, a number of competitive methods have tackled unsupervised repr...

Please sign up or login with your details

Forgot password? Click here to reset