Learning Contextualized Semantics from Co-occurring Terms via a Siamese Architecture

by   Ubai Sandouk, et al.

One of the biggest challenges in Multimedia information retrieval and understanding is to bridge the semantic gap by properly modeling concept semantics in context. The presence of out of vocabulary (OOV) concepts exacerbates this difficulty. To address the semantic gap issues, we formulate a problem on learning contextualized semantics from descriptive terms and propose a novel Siamese architecture to model the contextualized semantics from descriptive terms. By means of pattern aggregation and probabilistic topic models, our Siamese architecture captures contextualized semantics from the co-occurring descriptive terms via unsupervised learning, which leads to a concept embedding space of the terms in context. Furthermore, the co-occurring OOV concepts can be easily represented in the learnt concept embedding space. The main properties of the concept embedding space are demonstrated via visualization. Using various settings in semantic priming, we have carried out a thorough evaluation by comparing our approach to a number of state-of-the-art methods on six annotation corpora in different domains, i.e., MagTag5K, CAL500 and Million Song Dataset in the music domain as well as Corel5K, LabelMe and SUNDatabase in the image domain. Experimental results on semantic priming suggest that our approach outperforms those state-of-the-art methods considerably in various aspects.



There are no comments yet.


page 1

page 2

page 3

page 4


Measuring Semantic Relatedness using Mined Semantic Analysis

Mined Semantic Analysis (MSA) is a novel concept space model which emplo...

Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence

Visual-semantic embedding aims to learn a joint embedding space where re...

Evaluation of Siamese Networks for Semantic Code Search

With the increase in the number of open repositories and discussion foru...

Learning Modality-Invariant Representations for Speech and Images

In this paper, we explore the unsupervised learning of a semantic embedd...

Learnable Embedding Space for Efficient Neural Architecture Compression

We propose a method to incrementally learn an embedding space over the d...

Bridging the Gap between Semantics and Multimedia Processing

In this paper, we give an overview of the semantic gap problem in multim...

Automatic Spatially-aware Fashion Concept Discovery

This paper proposes an automatic spatially-aware concept discovery appro...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.