DeepAI AI Chat
Log In Sign Up

Learning Abstract Representations through Lossy Compression of Multi-Modal Signals

by   Charles Wilmot, et al.

A key competence for open-ended learning is the formation of increasingly abstract representations useful for driving complex behavior. Abstract representations ignore specific details and facilitate generalization. Here we consider the learning of abstract representations in a multi-modal setting with two or more input modalities. We treat the problem as a lossy compression problem and show that generic lossy compression of multimodal sensory input naturally extracts abstract representations that tend to strip away modalitiy specific details and preferentially retain information that is shared across the different modalities. Furthermore, we propose an architecture to learn abstract representations by identifying and retaining only the information that is shared across multiple modalities while discarding any modality specific information.


page 3

page 8


Robust Latent Representations via Cross-Modal Translation and Alignment

Multi-modal learning relates information across observation modalities o...

Private-Shared Disentangled Multimodal VAE for Learning of Hybrid Latent Representations

Multi-modal generative models represent an important family of deep mode...

SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities

As deep learning advances, there is an ever-growing demand for models ca...

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

Despite the remarkable success of deep multi-modal learning in practice,...

Factorized Multi-Modal Topic Model

Multi-modal data collections, such as corpora of paired images and text ...

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

As an important variant of entity alignment (EA), multi-modal entity ali...

Multimodal Deep Learning

This book is the result of a seminar in which we reviewed multimodal app...