Visually Grounded Neural Syntax Acquisition

06/07/2019
by   Haoyue Shi, et al.
2

We present the Visually Grounded Neural Syntax Learner (VG-NSL), an approach for learning syntactic representations and structures without any explicit supervision. The model learns by looking at natural images and reading paired captions. VG-NSL generates constituency parse trees of texts, recursively composes representations for constituents, and matches them with images. We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text. Experiments on the MSCOCO data set show that VG-NSL outperforms various unsupervised parsing approaches that do not use visual grounding, in terms of F1 scores against gold parse trees. We find that VGNSL is much more stable with respect to the choice of random initialization and the amount of training data. We also find that the concreteness acquired by VG-NSL correlates well with a similar measure defined by linguists. Finally, we also apply VG-NSL to multiple languages in the Multi30K data set, showing that our model consistently outperforms prior unsupervised approaches.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/04/2020

What is Learned in Visually Grounded Neural Syntax Acquisition

Visual features are a promising signal for learning bootstrap textual mo...
research
09/29/2021

Visually Grounded Concept Composition

We investigate ways to compose complex concepts in texts from primitive ...
research
06/05/2019

An Imitation Learning Approach to Unsupervised Parsing

Recently, there has been an increasing interest in unsupervised parsers ...
research
05/18/2018

SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text

Linguistic style is an essential part of written communication, with the...
research
04/10/2017

Automatic semantic role labeling on non-revised syntactic trees of journalistic texts

Semantic Role Labeling (SRL) is a Natural Language Processing task that ...
research
02/17/2022

Grammar-Based Grounded Lexicon Learning

We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist ...
research
04/28/2021

Learning Syntax from Naturally-Occurring Bracketings

Naturally-occurring bracketings, such as answer fragments to natural lan...

Please sign up or login with your details

Forgot password? Click here to reset