Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series

05/30/2019
by   Zhi-Xuan Tan, et al.
0

Integrating deep learning with latent state space models has the potential to yield temporal models that are powerful, yet tractable and interpretable. Unfortunately, current models are not designed to handle missing data or multiple data modalities, which are both prevalent in real-world data. In this work, we introduce a factorized inference method for Multimodal Deep Markov Models (MDMMs), allowing us to filter and smooth in the presence of missing data, while also performing uncertainty-aware multimodal fusion. We derive this method by factorizing the posterior p(z|x) for non-linear state space models, and develop a variational backward-forward algorithm for inference. Because our method handles incompleteness over both time and modalities, it is capable of interpolation, extrapolation, conditional generation, and label prediction in multimodal time series. We demonstrate these capabilities on both synthetic and real-world multimodal data under high levels of data deletion. Our method performs well even with more than 50 deep approaches to inference in latent time series.

READ FULL TEXT

page 8

page 16

research
06/11/2020

Stanza: A Nonlinear State Space Model for Probabilistic Inference in Non-Stationary Time Series

Time series with long-term structure arise in a variety of contexts and ...
research
02/25/2018

Time Series Analysis via Matrix Estimation

We consider the task of interpolating and forecasting a time series in t...
research
02/13/2020

Variational Conditional-Dependence Hidden Markov Models for Human Action Recognition

Hidden Markov Models (HMMs) are a powerful generative approach for model...
research
11/16/2022

Real Estate Attribute Prediction from Multiple Visual Modalities with Missing Data

The assessment and valuation of real estate requires large datasets with...
research
04/29/2021

A Randomized Missing Data Approach to Robust Filtering and Forecasting

We put forward a simple new randomized missing data (RMD) approach to ro...
research
11/22/2019

Differentiable Algorithm for Marginalising Changepoints

We present an algorithm for marginalising changepoints in time-series mo...
research
11/16/2021

Switching Recurrent Kalman Networks

Forecasting driving behavior or other sensor measurements is an essentia...

Please sign up or login with your details

Forgot password? Click here to reset