A CNN-RNN Framework for Image Annotation from Visual Cues and Social Network Metadata

10/13/2019
by   Tobia Tesan, et al.
52

Images represent a commonly used form of visual communication among people. Nevertheless, image classification may be a challenging task when dealing with unclear or non-common images needing more context to be correctly annotated. Metadata accompanying images on social-media represent an ideal source of additional information for retrieving proper neighbourhoods easing image annotation task. To this end, we blend visual features extracted from neighbours and their metadata to jointly leverage context and visual cues. Our models use multiple semantic embeddings to properly map metadata to a meaningful semantic space decoupling the neural model from the low-level representation of metadata and achieve robustness to vocabulary changes between training and testing phases. Convolutional and recurrent neural networks (CNNs-RNNs) are jointly adopted to infer similarity among neighbours and query images. We perform comprehensive experiments on the NUS-WIDE dataset showing that our models outperform state-of-the-art architectures based on images and metadata, and decrease both sensory and semantic gaps to better annotate images.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/30/2015

Love Thy Neighbors: Image Annotation by Exploiting Image Metadata

Some images that are difficult to recognize on their own may become more...
research
01/27/2018

Deep Neural Networks In Fully Connected CRF For Image Labeling With Social Network Metadata

We propose a novel method for predicting image labels by fusing image co...
research
07/16/2012

Image Labeling on a Network: Using Social-Network Metadata for Image Classification

Large-scale image retrieval benchmarks invariably consist of images from...
research
03/02/2019

AIRD: Adversarial Learning Framework for Image Repurposing Detection

Image repurposing is a commonly used method for spreading misinformation...
research
06/19/2017

Exploring Content-based Artwork Recommendation with Metadata and Visual Features

Compared to other areas, artwork recommendation has received little atte...
research
01/05/2018

VSE-ens: Visual-Semantic Embeddings with Efficient Negative Sampling

Jointing visual-semantic embeddings (VSE) have become a research hotpot ...
research
08/18/2023

Metadata Improves Segmentation Through Multitasking Elicitation

Metainformation is a common companion to biomedical images. However, thi...

Please sign up or login with your details

Forgot password? Click here to reset