Can neural networks acquire a structural bias from raw linguistic data?

07/14/2020
by   Alex Warstadt, et al.
0

We evaluate whether BERT, a widely used neural network for sentence processing, acquires an inductive bias towards forming structural generalizations through pretraining on raw data. We conduct four experiments testing its preference for structural vs. linear generalizations in different structure-dependent phenomena. We find that BERT makes a structural generalization in 3 out of 4 empirical domains—subject-auxiliary inversion, reflexive binding, and verb tense detection in embedded clauses—but makes a linear generalization when tested on NPI licensing. We argue that these results are the strongest evidence so far from artificial learners supporting the proposition that a structural bias can be acquired from raw data. If this conclusion is correct, it is tentative evidence that some linguistic universals can be acquired by learners without innate biases. However, the precise implications for human language acquisition are unclear, as humans learn language from significantly less data than BERT.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/17/2022

What Artificial Neural Networks Can Tell Us About Human Language Acquisition

Rapid progress in machine learning for natural language processing has t...
06/29/2020

Universal linguistic inductive biases via meta-learning

How do learners acquire languages from the limited data available to the...
06/26/2020

What they do when in doubt: a study of inductive biases in seq2seq learners

Sequence-to-sequence (seq2seq) learners are widely used, but we still ha...
02/25/2018

Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks

Syntactic rules in human language usually refer to the hierarchical stru...
05/31/2018

Neural Network Acceptability Judgments

In this work, we explore the ability of artificial neural networks to ju...
10/08/2021

Distinguishing rule- and exemplar-based generalization in learning systems

Despite the increasing scale of datasets in machine learning, generaliza...
09/23/2021

Revisiting the Uniform Information Density Hypothesis

The uniform information density (UID) hypothesis posits a preference amo...