DeepAI AI Chat
Log In Sign Up

Learning Abstract Representations through Lossy Compression of Multi-Modal Signals

01/27/2021
by   Charles Wilmot, et al.
0

A key competence for open-ended learning is the formation of increasingly abstract representations useful for driving complex behavior. Abstract representations ignore specific details and facilitate generalization. Here we consider the learning of abstract representations in a multi-modal setting with two or more input modalities. We treat the problem as a lossy compression problem and show that generic lossy compression of multimodal sensory input naturally extracts abstract representations that tend to strip away modalitiy specific details and preferentially retain information that is shared across the different modalities. Furthermore, we propose an architecture to learn abstract representations by identifying and retaining only the information that is shared across multiple modalities while discarding any modality specific information.

READ FULL TEXT

page 3

page 8

11/03/2020

Robust Latent Representations via Cross-Modal Translation and Alignment

Multi-modal learning relates information across observation modalities o...
12/23/2020

Private-Shared Disentangled Multimodal VAE for Learning of Hybrid Latent Representations

Multi-modal generative models represent an important family of deep mode...
04/30/2022

SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities

As deep learning advances, there is an ever-growing demand for models ca...
03/23/2022

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

Despite the remarkable success of deep multi-modal learning in practice,...
10/16/2012

Factorized Multi-Modal Topic Model

Multi-modal data collections, such as corpora of paired images and text ...
12/29/2022

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

As an important variant of entity alignment (EA), multi-modal entity ali...
01/12/2023

Multimodal Deep Learning

This book is the result of a seminar in which we reviewed multimodal app...