Towards Visual Semantics

04/26/2021
by   Fausto Giunchiglia, et al.
0

In Visual Semantics we study how humans build mental representations, i.e., concepts , of what they visually perceive. We call such concepts, substance concepts. In this paper we provide a theory and an algorithm which learns substance concepts which correspond to the concepts, that we call classification concepts , that in Lexical Semantics are used to encode word meanings. The theory and algorithm are based on three main contributions: (i) substance concepts are modeled as visual objects , namely sequences of similar frames, as perceived in multiple encounters ; (ii) substance concepts are organized into a visual subsumption hierarchy based on the notions of Genus and Differentia that resemble the notions that, in Lexical Semantics, allow to construct hierarchies of classification concepts; (iii) the human feedback is exploited not to name objects, as it has been the case so far, but, rather, to align the hierarchy of substance concepts with that of classification concepts. The learning algorithm is implemented for the base case of a hierarchy of depth two. The experiments, though preliminary, show that the algorithm manages to acquire the notions of Genus and Differentia with reasonable accuracy, this despite seeing a small number of examples and receiving supervision on a fraction of them.

READ FULL TEXT

page 19

page 20

research
12/13/2022

Aligning Visual and Lexical Semantics

We discuss two kinds of semantics relevant to Computer Vision (CV) syste...
research
05/19/2021

Classifying concepts via visual properties

We assume that substances in the world are represented by two types of c...
research
10/25/2019

Is it a Fruit, an Apple or a Granny Smith? Predicting the Basic Level in a Concept Hierarchy

The "basic level", according to experiments in cognitive psychology, is ...
research
02/26/2022

Building a visual semantics aware object hierarchy

The semantic gap is defined as the difference between the linguistic rep...
research
05/15/2022

Discovering Latent Concepts Learned in BERT

A large number of studies that analyze deep neural network models and th...
research
07/05/2022

Pretraining on Interactions for Learning Grounded Affordance Representations

Lexical semantics and cognitive science point to affordances (i.e. the a...

Please sign up or login with your details

Forgot password? Click here to reset