EMISSOR: A platform for capturing multimodal interactions as Episodic Memories and Interpretations with Situated Scenario-based Ontological References

05/18/2021
by   Selene Báez Santamaría, et al.
0

We present EMISSOR: a platform to capture multimodal interactions as recordings of episodic experiences with explicit referential interpretations that also yield an episodic Knowledge Graph (eKG). The platform stores streams of multiple modalities as parallel signals. Each signal is segmented and annotated independently with interpretation. Annotations are eventually mapped to explicit identities and relations in the eKG. As we ground signal segments from different modalities to the same instance representations, we also ground different modalities across each other. Unique to our eKG is that it accepts different interpretations across modalities, sources and experiences and supports reasoning over conflicting information and uncertainties that may result from multimodal experiences. EMISSOR can record and annotate experiments in virtual and real-world, combine data, evaluate system behavior and their performance for preset goals but also model the accumulation of knowledge and interpretations in the Knowledge Graph as a result of these episodic experiences.

READ FULL TEXT

page 8

page 9

page 13

page 14

page 15

page 16

page 18

page 19

research
06/01/2022

A modular architecture for creating multimodal agents

The paper describes a flexible and modular platform to create multimodal...
research
08/20/2023

Imaginations of WALL-E : Reconstructing Experiences with an Imagination-Inspired Module for Advanced AI Systems

In this paper, we introduce a novel Artificial Intelligence (AI) system ...
research
06/08/2021

What Makes Multimodal Learning Better than Single (Provably)

The world provides us with data of multiple modalities. Intuitively, mod...
research
05/15/2023

Continual Multimodal Knowledge Graph Construction

Multimodal Knowledge Graph Construction (MKGC) involves creating structu...
research
02/14/2022

Close-up and Whispering: An Understanding of Multimodal and Parasocial Interactions in YouTube ASMR videos

ASMR (Autonomous Sensory Meridian Response) has grown to immense popular...
research
07/19/2023

Estudio de la Experiencia de Usuario mediante un Sistema de Dashboards de Análisis de Aprendizaje Multimodal

In the article, we present a Web-based System called M2LADS, which suppo...
research
03/06/2020

Knowledge graph based methods for record linkage

Nowadays, it is common in Historical Demography the use of individual-le...

Please sign up or login with your details

Forgot password? Click here to reset