The Limitations of Limited Context for Constituency Parsing

06/03/2021
by   Yuchen Li, et al.
6

Incorporating syntax into neural approaches in NLP has a multitude of practical and scientific benefits. For instance, a language model that is syntax-aware is likely to be able to produce better samples; even a discriminative model like BERT with a syntax module could be used for core NLP tasks like unsupervised syntactic parsing. Rapid progress in recent years was arguably spurred on by the empirical success of the Parsing-Reading-Predict architecture of (Shen et al., 2018a), later simplified by the Order Neuron LSTM of (Shen et al., 2019). Most notably, this is the first time neural approaches were able to successfully perform unsupervised syntactic parsing (evaluated by various metrics like F-1 score). However, even heuristic (much less fully mathematical) understanding of why and when these architectures work is lagging severely behind. In this work, we answer representational questions raised by the architectures in (Shen et al., 2018a, 2019), as well as some transition-based syntax-aware language models (Dyer et al., 2016): what kind of syntactic structure can current neural approaches to syntax represent? Concretely, we ground this question in the sandbox of probabilistic context-free-grammars (PCFGs), and identify a key aspect of the representational power of these approaches: the amount and directionality of context that the predictor has access to when forced to make parsing decision. We show that with limited context (either bounded, or unidirectional), there are PCFGs, for which these approaches cannot represent the max-likelihood parse; conversely, if the context is unlimited, they can represent the max-likelihood parse of any PCFG.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/10/2022

Unsupervised and Few-shot Parsing from Pretrained Language Models

Pretrained language models are generally acknowledged to be able to enco...
research
04/28/2021

Learning Syntax from Naturally-Occurring Bracketings

Naturally-occurring bracketings, such as answer fragments to natural lan...
research
10/05/2020

A Pilot Study of Text-to-SQL Semantic Parsing for Vietnamese

Semantic parsing is an important NLP task. However, Vietnamese is a low-...
research
09/20/2019

A Critical Analysis of Biased Parsers in Unsupervised Parsing

A series of recent papers has used a parsing algorithm due to Shen et al...
research
11/12/2018

Syntax Helps ELMo Understand Semantics: Is Syntax Still Relevant in a Deep Neural Architecture for SRL?

Do unsupervised methods for learning rich, contextualized token represen...
research
05/04/2020

What is Learned in Visually Grounded Neural Syntax Acquisition

Visual features are a promising signal for learning bootstrap textual mo...
research
06/05/2019

An Imitation Learning Approach to Unsupervised Parsing

Recently, there has been an increasing interest in unsupervised parsers ...

Please sign up or login with your details

Forgot password? Click here to reset