The Curious Case of Control

05/24/2022
by   Elias Stengel-Eskin, et al.
0

Children acquiring English make systematic errors on subject control sentences even after they have reached near-adult competence (C. Chomsky, 1969), possibly due to heuristics based on semantic roles (Maratsos, 1974). Given the advanced fluency of large generative language models, we ask whether model outputs are consistent with these heuristics, and to what degree different models are consistent with each other. We find that models can be categorized by behavior into three separate groups, with broad differences between the groups. The outputs of models in the largest group are consistent with positional heuristics that succeed on subject control but fail on object control. This result is surprising, given that object control is orders of magnitude more frequent in the text data used to train such models. We examine to what degree the models are sensitive to prompting with agent-patient information, finding that raising the salience of agent and patient relations results in significant changes in the outputs of most models. Based on this observation, we leverage an existing dataset of semantic proto-role annotations (White, et al. 2020) to explore the connections between control and labeling event participants with properties typically associated with agents and patients.

READ FULL TEXT
research
11/10/2022

Measuring Reliability of Large Language Models through Semantic Consistency

While large pretrained language models (PLMs) demonstrate incredible flu...
research
06/23/2022

Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models

NLP models trained on text have been shown to reproduce human stereotype...
research
04/13/2021

Detoxifying Language Models Risks Marginalizing Minority Voices

Language models (LMs) must be both safe and equitable to be responsibly ...
research
02/04/2019

An Argument-Marker Model for Syntax-Agnostic Proto-Role Labeling

Semantic proto-role labeling (SPRL) is an alternative to semantic role l...
research
06/24/2019

A Logic-Based Learning Approach to Explore Diabetes Patient Behaviors

Type I Diabetes (T1D) is a chronic disease in which the body's ability t...
research
08/23/2023

Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models

AI programs, built using large language models, make it possible to auto...
research
07/01/2021

The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models

Supervised learning models often make systematic errors on rare subsets ...

Please sign up or login with your details

Forgot password? Click here to reset