Cultural and Geographical Influences on Image Translatability of Words across Languages

by   Isidora Chara Tourni, et al.

Neural Machine Translation (NMT) models have been observed to produce poor translations when there are few/no parallel sentences to train the models. In the absence of parallel data, several approaches have turned to the use of images to learn translations. Since images of words, e.g., horse may be unchanged across languages, translations can be identified via images associated with words in different languages that have a high degree of visual similarity. However, translating via images has been shown to improve upon text-only models only marginally. To better understand when images are useful for translation, we study image translatability of words, which we define as the translatability of words via images, by measuring intra- and inter-cluster similarities of image representations of words that are translations of each other. We find that images of words are not always invariant across languages, and that language pairs with shared culture, meaning having either a common language family, ethnicity or religion, have improved image translatability (i.e., have more similar images for similar words) compared to its converse, regardless of their geographic proximity. In addition, in line with previous works that show images help more in translating concrete words, we found that concrete words have improved image translatability compared to abstract ones.



There are no comments yet.


page 8

page 9


From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots

The neural machine translation model has suffered from the lack of large...

Paraphrases as Foreign Languages in Multilingual Neural Machine Translation

Using paraphrases, the expression of the same semantic meaning in differ...

Bag-of-Words as Target for Neural Machine Translation

A sentence can be translated into more than one correct sentences. Howev...

An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification

End-to-end neural machine translation has overtaken statistical machine ...

Bridging Neural Machine Translation and Bilingual Dictionaries

Neural Machine Translation (NMT) has become the new state-of-the-art in ...

Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data

Bilingual lexicon induction, translating words from the source language ...

The artificial synesthete: Image-melody translations with variational autoencoders

Abstract This project presents a system of neural networks to translate ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.