FusionVAE: A Deep Hierarchical Variational Autoencoder for RGB Image Fusion

09/22/2022
by   Fabian Duffhauss, et al.
0

Sensor fusion can significantly improve the performance of many computer vision tasks. However, traditional fusion approaches are either not data-driven and cannot exploit prior knowledge nor find regularities in a given dataset or they are restricted to a single application. We overcome this shortcoming by presenting a novel deep hierarchical variational autoencoder called FusionVAE that can serve as a basis for many fusion tasks. Our approach is able to generate diverse image samples that are conditioned on multiple noisy, occluded, or only partially visible input images. We derive and optimize a variational lower bound for the conditional log-likelihood of FusionVAE. In order to assess the fusion capabilities of our model thoroughly, we created three novel datasets for image fusion based on popular computer vision datasets. In our experiments, we show that FusionVAE learns a representation of aggregated information that is relevant to fusion tasks. The results demonstrate that our approach outperforms traditional methods significantly. Furthermore, we present the advantages and disadvantages of different design choices.

READ FULL TEXT

page 11

page 12

page 21

page 24

page 25

research
04/16/2020

Conditioned Variational Autoencoder for top-N item recommendation

In this paper, we propose a Conditioned Variational Autoencoder to impro...
research
03/18/2019

M^2VAE - Derivation of a Multi-Modal Variational Autoencoder Objective from the Marginal Joint Log-Likelihood

This work gives an in-depth derivation of the trainable evidence lower b...
research
09/25/2020

Hierarchical Sparse Variational Autoencoder for Text Encoding

In this paper we focus on unsupervised representation learning and propo...
research
10/07/2019

Negative Sampling in Variational Autoencoders

We propose negative sampling as an approach to improve the notoriously b...
research
06/15/2023

Tree Variational Autoencoders

We propose a new generative hierarchical clustering model that learns a ...
research
07/09/2023

Visible and infrared self-supervised fusion trained on a single example

This paper addresses the problem of visible (RGB) to Near-Infrared (NIR)...
research
10/26/2022

Fast and Efficient Scene Categorization for Autonomous Driving using VAEs

Scene categorization is a useful precursor task that provides prior know...

Please sign up or login with your details

Forgot password? Click here to reset