A Visuospatial Dataset for Naturalistic Verb Learning

10/28/2020
by   Dylan Ebert, et al.
0

We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality environment and is designed to emulate the quality of language data to which a pre-verbal child is likely to have access: That is, naturalistic, spontaneous speech paired with richly grounded visuospatial context. We use the collected data to compare several distributional semantics models for verb learning. We evaluate neural models based on 2D (pixel) features as well as feature-engineered models based on 3D (symbolic, spatial) features, and show that neither modeling approach achieves satisfactory performance. Our results are consistent with evidence from child language acquisition that emphasizes the difficulty of learning verbs from naive distributional data. We discuss avenues for future work on cognitively-inspired grounded language learning, and release our corpus with the intent of facilitating research on the topic.

READ FULL TEXT

page 2

page 8

research
04/22/2022

Learning Functional Distributional Semantics with Visual Data

Functional Distributional Semantics is a recently proposed framework for...
research
12/27/2021

Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech

Learning to understand grounded language, which connects natural languag...
research
07/26/2016

Grounded Lexicon Acquisition - Case Studies in Spatial Language

This paper discusses grounded acquisition experiments of increasing comp...
research
08/30/2023

Benchmarking Multilabel Topic Classification in the Kyrgyz Language

Kyrgyz is a very underrepresented language in terms of modern natural la...
research
04/26/2023

Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models

Current dialogue research primarily studies pairwise (two-party) convers...
research
07/20/2021

Neural Variational Learning for Grounded Language Acquisition

We propose a learning system in which language is grounded in visual per...
research
12/11/2019

Just Add Functions: A Neural-Symbolic Language Model

Neural network language models (NNLMs) have achieved ever-improving accu...

Please sign up or login with your details

Forgot password? Click here to reset