Image Pivoting for Learning Multilingual Multimodal Representations

07/24/2017
by   Spandana Gella, et al.
0

In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image understanding. Our model learns a common representation for images and their descriptions in two different languages (which need not be parallel) by considering the image as a pivot between two languages. We introduce a new pairwise ranking loss function which can handle both symmetric and asymmetric similarity between the two modalities. We evaluate our models on image-description ranking for German and English, and on semantic textual similarity of image descriptions in English. In both cases we achieve state-of-the-art performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/30/2019

Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations

With the aim of promoting and understanding the multilingual version of ...
research
05/02/2016

Multi30K: Multilingual English-German Image Descriptions

We introduce the Multi30K dataset to stimulate multilingual multimodal r...
research
11/09/2019

Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning

Recent work has highlighted the advantage of jointly learning grounded s...
research
02/03/2017

Multilingual Multi-modal Embeddings for Natural Language Processing

We propose a novel discriminative model that learns embeddings from mult...
research
07/06/2017

Cross-linguistic differences and similarities in image descriptions

Automatic image description systems are commonly trained and evaluated o...
research
03/13/2017

A Visual Representation of Wittgenstein's Tractatus Logico-Philosophicus

In this paper we present a data visualization method together with its p...
research
10/13/2015

Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

Recently there has been a lot of interest in learning common representat...

Please sign up or login with your details

Forgot password? Click here to reset