Learning to Learn Words from Narrated Video

11/25/2019
by   Dídac Surís, et al.
12

When we travel, we often encounter new scenarios we have never experienced before, with new sights and new words that describe them. We can use our language-learning ability to quickly learn these new words and correlate them with the visual world. In contrast, language models often do not robustly generalize to novel words and compositions. We propose a framework that learns how to learn text representations from visual context. Experiments show that our approach significantly outperforms the state-of-the-art in visual language modeling for acquiring new words and predicting new compositions. Model ablations and visualizations suggest that the visual modality helps our approach more robustly generalize at these tasks. Project webpage is available at https://expert.cs.columbia.edu/

READ FULL TEXT

page 1

page 2

page 3

page 7

page 8

page 9

research
05/29/2018

Visually Grounded, Situated Learning in Neural Models

The theory of situated cognition postulates that language is inseparable...
research
06/14/2023

World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models

The ability to connect language units to their referents in the physical...
research
10/29/2018

Language Modeling with Sparse Product of Sememe Experts

Most language modeling methods rely on large-scale data to statistically...
research
08/04/2020

Learning Visual Representations with Caption Annotations

Pretraining general-purpose visual features has become a crucial part of...
research
04/17/2023

Towards Robust Prompts on Vision-Language Models

With the advent of vision-language models (VLMs) that can perform in-con...
research
01/26/2023

Affective Faces for Goal-Driven Dyadic Communication

We introduce a video framework for modeling the association between verb...
research
06/30/2020

Modality-Agnostic Attention Fusion for visual search with text feedback

Image retrieval with natural language feedback offers the promise of cat...

Please sign up or login with your details

Forgot password? Click here to reset