Multimodal Sensor Fusion with Differentiable Filters

10/25/2020
by   Michelle A. Lee, et al.
0

Leveraging multimodal information with recursive Bayesian filters improves performance and robustness of state estimation, as recursive filters can combine different modalities according to their uncertainties. Prior work has studied how to optimally fuse different sensor modalities with analytical state estimation algorithms. However, deriving the dynamics and measurement models along with their noise profile can be difficult or lead to intractable models. Differentiable filters provide a way to learn these models end-to-end while retaining the algorithmic structure of recursive filters. This can be especially helpful when working with sensor modalities that are high dimensional and have very different characteristics. In contact-rich manipulation, we want to combine visual sensing (which gives us global information) with tactile sensing (which gives us local information). In this paper, we study new differentiable filtering architectures to fuse heterogeneous sensor information. As case studies, we evaluate three tasks: two in planar pushing (simulated and real) and one in manipulating a kinematically constrained door (simulated). In extensive evaluations, we find that differentiable filters that leverage crossmodal sensor information reach comparable accuracies to unstructured LSTM models, while presenting interpretability benefits that may be important for safety-critical systems. We also release an open-source library for creating and training differentiable Bayesian filters in PyTorch, which can be found on our project website: https://sites.google.com/view/ multimodalfilter.

READ FULL TEXT

page 1

page 5

page 7

05/28/2018

Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors

We present differentiable particle filters (DPFs): a differentiable impl...
12/28/2020

How to Train Your Differentiable Filter

In many robotic applications, it is crucial to maintain a belief about t...
05/18/2021

Differentiable Factor Graph Optimization for Learning Smoothers

A recent line of work has shown that end-to-end optimization of Bayesian...
04/18/2022

Multimodal Proximity and Visuotactile Sensing With a Selectively Transmissive Soft Membrane

The most common sensing modalities found in a robot perception system ar...
11/11/2020

End-To-End Semi-supervised Learning for Differentiable Particle Filters

Recent advances in incorporating neural networks into particle filters p...
04/06/2021

Out-of-Distribution Robustness with Deep Recursive Filters

Accurate state and uncertainty estimation is imperative for mobile robot...