Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games

by   Alessandro Suglia, et al.

In visual guessing games, a Guesser has to identify a target object in a scene by asking questions to an Oracle. An effective strategy for the players is to learn conceptual representations of objects that are both discriminative and expressive enough to ask questions and guess correctly. However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at training and inference time. This provides an unnatural performance advantage when categories at inference time match those at training time, and it causes models to fail in more realistic "zero-shot" scenarios where out-of-domain object categories are involved. To overcome this issue, we introduce a novel "imagination" module based on Regularized Auto-Encoders, that learns context-aware and category-aware latent embeddings without relying on category labels at inference time. Our imagination module outperforms state-of-the-art competitors by 8.26 zero-shot scenario (Suglia et al., 2020), and it improves the Oracle and Guesser accuracy by 2.08 categories are available at inference time. The imagination module also boosts reasoning about object properties and attributes.


page 2

page 8


Zero Shot Recognition with Unreliable Attributes

In principle, zero-shot learning makes it possible to train a recognitio...

CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning

Approaches to Grounded Language Learning typically focus on a single tas...

Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories

Zero-shot learning in Language & Vision is the task of correctly labelli...

Zero-shot object prediction using semantic scene knowledge

This work focuses on the semantic relations between scenes and objects f...

Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs

We consider the problem of zero-shot recognition: learning a visual clas...

Modelling Semantic Categories using Conceptual Neighborhood

While many methods for learning vector space embeddings have been propos...

Finding any Waldo: zero-shot invariant and efficient visual search

Searching for a target object in a cluttered scene constitutes a fundame...