An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models

04/06/2020
by   Hiroshi Noji, et al.
0

We explore the utilities of explicit negative examples in training neural language models. Negative examples here are incorrect words in a sentence, such as "barks" in "*The dogs barks". Neural language models are commonly trained only on positive examples, a set of sentences in the training data, but recent studies suggest that the models trained in this way are not capable of robustly handling complex syntactic constructions, such as long-distance agreement. In this paper, using English data, we first demonstrate that appropriately using negative examples about particular constructions (e.g., subject-verb agreement) will boost the model's robustness on them, with a negligible loss of perplexity. The key to our success is an additional margin loss between the log-likelihoods of a correct word and an incorrect word. We then provide a detailed analysis of the trained models. One of our findings is the difficulty of object-relative clauses for RNNs. We find that even with our direct learning signals the models still suffer from resolving agreement across an object-relative clause. Augmentation of training sentences involving the constructions somewhat helps, but the accuracy still does not reach the level of subject-relative clauses. Although not directly cognitively appealing, our method can be a tool to analyze the true architectural limitation of neural models on challenging linguistic constructions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/28/2022

Probing for targeted syntactic knowledge through grammatical error detection

Targeted studies testing knowledge of subject-verb agreement (SVA) indic...
research
06/12/2017

Exploring the Syntactic Abilities of RNNs with Multi-task Learning

Recent work has explored the syntactic abilities of RNNs using the subje...
research
06/10/2021

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models

Targeted syntactic evaluations have demonstrated the ability of language...
research
05/01/2020

Cross-Linguistic Syntactic Evaluation of Word Prediction Models

A range of studies have concluded that neural word prediction models can...
research
05/01/2020

Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

A standard approach to evaluating language models analyzes how models as...
research
05/29/2023

A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces

We study semantic construal in grammatical constructions using large lan...
research
08/31/2018

Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items

In this paper, we attempt to link the inner workings of a neural languag...

Please sign up or login with your details

Forgot password? Click here to reset