Syntactic Substitutability as Unsupervised Dependency Syntax

11/29/2022
by   Jasper Jian, et al.
0

Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. An active line of inquiry is whether large pretrained language models (LLMs) are able to acquire syntax by training on text alone; understanding a model's syntactic capabilities is essential to understanding how it processes and makes use of language. In this paper, we propose a new method, SSUD, which allows for the induction of syntactic structures without supervision from gold-standard parses. Instead, we seek to define formalism-agnostic, model-intrinsic syntactic parses by using a property of syntactic relations: syntactic substitutability. We demonstrate both quantitative and qualitative gains on dependency parsing tasks using SSUD, and induce syntactic structures which we hope provide clarity into LLMs and linguistic representations, alike.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/30/2019

Parsing All: Syntax and Semantics, Dependencies and Spans

Both syntactic and semantic structures are key linguistic contextual clu...
research
05/11/2018

Deep RNNs Encode Soft Hierarchical Syntax

We present a set of experiments to demonstrate that deep recurrent neura...
research
09/13/2023

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

Most interpretability research in NLP focuses on understanding the behav...
research
12/05/2021

The Linear Arrangement Library. A new tool for research on syntactic dependency structures

The new and growing field of Quantitative Dependency Syntax has emerged ...
research
02/17/2023

False perspectives on human language: why statistics needs linguistics

A sharp tension exists about the nature of human language between two op...
research
10/27/2022

Natural Language Syntax Complies with the Free-Energy Principle

Natural language syntax yields an unbounded array of hierarchically stru...
research
09/20/2023

Assessment of Pre-Trained Models Across Languages and Grammars

We present an approach for assessing how multilingual large language mod...

Please sign up or login with your details

Forgot password? Click here to reset