Learning more expressive joint distributions in multimodal variational methods

09/08/2020
by   Sasho Nedelkoski, et al.
0

Data often are formed of multiple modalities, which jointly describe the observed phenomena. Modeling the joint distribution of multimodal data requires larger expressive power to capture high-level concepts and provide better data representations. However, multimodal generative models based on variational inference are limited due to the lack of flexibility of the approximate posterior, which is obtained by searching within a known parametric family of distributions. We introduce a method that improves the representational capacity of multimodal variational methods using normalizing flows. It approximates the joint posterior with a simple parametric distribution and subsequently transforms into a more complex one. Through several experiments, we demonstrate that the model improves on state-of-the-art multimodal methods based on variational inference on various computer vision tasks such as colorization, edge and mask detection, and weakly supervised learning. We also show that learning more powerful approximate joint distributions improves the quality of the generated samples. The code of our model is publicly available at https://github.com/SashoNedelkoski/BPFDMVM.

READ FULL TEXT

page 8

page 10

02/14/2018

Multimodal Generative Models for Scalable Weakly-Supervised Learning

Multiple modalities often co-occur when describing natural phenomena. Le...
05/21/2015

Variational Inference with Normalizing Flows

The choice of approximate posterior distribution is one of the core prob...
06/06/2019

Amortized Inference of Variational Bounds for Learning Noisy-OR

Classical approaches for approximate inference depend on cleverly design...
03/22/2022

Self-Supervised Representation Learning as Multimodal Variational Inference

This paper proposes a probabilistic extension of SimSiam, a recent self-...
06/15/2020

Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence

Learning from different data types is a long-standing goal in machine le...
12/20/2021

Multimodal Adversarially Learned Inference with Factorized Discriminators

Learning from multimodal data is an important research topic in machine ...
05/06/2021

Generalized Multimodal ELBO

Multiple data types naturally co-occur when describing real-world phenom...