Image2song: Song Retrieval via Bridging Image Content and Lyric Words

08/19/2017
by   Xuelong Li, et al.
0

Image is usually taken for expressing some kinds of emotions or purposes, such as love, celebrating Christmas. There is another better way that combines the image and relevant song to amplify the expression, which has drawn much attention in the social network recently. Hence, the automatic selection of songs should be expected. In this paper, we propose to retrieve semantic relevant songs just by an image query, which is named as the image2song problem. Motivated by the requirements of establishing correlation in semantic/content, we build a semantic-based song retrieval framework, which learns the correlation between image content and lyric words. This model uses a convolutional neural network to generate rich tags from image regions, a recurrent neural network to model lyric, and then establishes correlation via a multi-layer perceptron. To reduce the content gap between image and lyric, we propose to make the lyric modeling focus on the main image content via a tag attention. We collect a dataset from the social-sharing multimodal data to study the proposed problem, which consists of (image, music clip, lyric) triplets. We demonstrate that our proposed model shows noticeable results in the image2song retrieval task and provides suitable songs. Besides, the song2image task is also performed.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 7

page 8

page 11

page 12

research
06/19/2017

Recent Advance in Content-based Image Retrieval: A Literature Survey

The explosive increase and ubiquitous accessibility of visual data on th...
research
03/02/2017

A novel image tag completion method based on convolutional neural network

In the problems of image retrieval and annotation, complete textual tag ...
research
02/28/2016

Measuring and Predicting Tag Importance for Image Retrieval

Textual data such as tags, sentence descriptions are combined with visua...
research
07/16/2012

Image Labeling on a Network: Using Social-Network Metadata for Image Classification

Large-scale image retrieval benchmarks invariably consist of images from...
research
09/05/2017

Predicting Visual Features from Text for Image and Video Caption Retrieval

This paper strives to find amidst a set of sentences the one best descri...
research
04/23/2015

Multimodal Convolutional Neural Networks for Matching Image and Sentence

In this paper, we propose multimodal convolutional neural networks (m-CN...
research
07/10/2023

Retrieval of phonemes and Kohonen algorithm

A phoneme-retrieval technique is proposed, which is due to the particula...

Please sign up or login with your details

Forgot password? Click here to reset