Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference

by   Yoshinobu Hagiwara, et al.

This paper describes a computational model of multiagent multimodal categorization that realizes emergent communication. We clarify whether the computational model can reproduce the following functions in a symbol emergence system, comprising two agents with different sensory modalities playing a naming game. (1) Function for forming a shared lexical system that comprises perceptual categories and corresponding signs, formed by agents through individual learning and semiotic communication between agents. (2) Function to improve the categorization accuracy in an agent via semiotic communication with another agent, even when some sensory modalities of each agent are missing. (3) Function that an agent infers unobserved sensory information based on a sign sampled from another agent in the same manner as cross-modal inference. We propose an interpersonal multimodal Dirichlet mixture (Inter-MDM), which is derived by dividing an integrative probabilistic generative model, which is obtained by integrating two Dirichlet mixtures (DMs). The Markov chain Monte Carlo algorithm realizes emergent communication. The experimental results demonstrated that Inter-MDM enables agents to form multimodal categories and appropriately share signs between agents. It is shown that emergent communication improves categorization accuracy, even when some sensory modalities are missing. Inter-MDM enables an agent to predict unobserved information based on a shared sign.



There are no comments yet.


page 2

page 16

page 20


Symbol Emergence as an Interpersonal Multimodal Categorization

This study focuses on category formation for individual agents and the d...

Symbol Emergence in Robotics: A Survey

Humans can learn the use of language through physical interaction with t...

Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learning

In this work we explore the use of latent representations obtained from ...

Multimodal representation models for prediction and control from partial information

Similar to humans, robots benefit from interacting with their environmen...

Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

Multimodal learning aims to discover the relationship between multiple m...

Neural Multisensory Scene Inference

For embodied agents to infer representations of the underlying 3D physic...

The Utility of Hedged Assertions in the Emergence of Shared Categorical Labels

We investigate the emergence of shared concepts in a community of langua...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.