DeepAI AI Chat
Log In Sign Up

Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

by   Danny Merkx, et al.
Radboud Universiteit

Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are purely text based, even though the human sensory experience is much richer. In this paper we create visually grounded word embeddings by combining English text and images and compare them to popular text-based methods, to see if visual information allows our model to better capture cognitive aspects of word meaning. Our analysis shows that visually grounded embedding similarities are more predictive of the human reaction times in a large priming experiment than the purely text-based embeddings. The visually grounded embeddings also correlate well with human word similarity ratings. Importantly, in both experiments we show that the grounded embeddings account for a unique portion of explained variance, even when we include text-based embeddings trained on huge corpora. This shows that visual grounding allows our model to capture information that cannot be extracted using text as the only source of information.


Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes

We propose a model to learn visually grounded word embeddings (vis-w2v) ...

comp-syn: Perceptually Grounded Word Embeddings with Color

Popular approaches to natural language processing create word embeddings...

Trick or TReAT: Thematic Reinforcement for Artistic Typography

An approach to make text visually appealing and memorable is semantic re...

Efficient Multi-Modal Embeddings from Structured Data

Multi-modal word semantics aims to enhance embeddings with perceptual in...

A Game Interface to Study Semantic Grounding in Text-Based Models

Can language models learn grounded representations from text distributio...

Sound-Word2Vec: Learning Word Representations Grounded in Sounds

To be able to interact better with humans, it is crucial for machines to...

Exploring the Sensory Spaces of English Perceptual Verbs in Natural Language Data

In this study, we explore how language captures the meaning of words, in...