DeepAI AI Chat
Log In Sign Up

Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images

by   Junhua Mao, et al.
Pinterest, Inc.

In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest. This dataset is more than 200 times larger than MS COCO, the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of finding semantically similar or related words and phrases. The word/phrase pairs in this evaluation dataset are collected from the click data with millions of users in an image search system, thus contain rich semantic relationships. Based on these datasets, we propose and compare several Recurrent Neural Networks (RNNs) based multimodal (text and image) models. Experiments show that our model benefits from incorporating the visual information into the word embeddings, and a weight sharing strategy is crucial for learning such multimodal embeddings. The project page is: junhua.mao/multimodal_embedding.html


Word Embeddings from Large-Scale Greek Web content

Word embeddings are undoubtedly very useful components in many NLP tasks...

Learning and Evaluating Sparse Interpretable Sentence Embeddings

Previous research on word embeddings has shown that sparse representatio...

Embedding Arithmetic for Text-driven Image Transformation

Latent text representations exhibit geometric regularities, such as the ...

Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching

Existing image-text matching approaches typically infer the similarity o...

One-Shot Item Search with Multimodal Data

In the task of near similar image search, features from Deep Neural Netw...

MTTN: Multi-Pair Text to Text Narratives for Prompt Generation

The increased interest in diffusion models has opened up opportunities f...