research
∙
08/07/2023
Studying Large Language Model Generalization with Influence Functions
When trying to gain better visibility into a machine learning model in o...
share
research
∙
06/15/2023
Inverse Scaling: When Bigger Isn't Better
Work on scaling laws has found that large language models (LMs) show pre...
research
∙
05/30/2023
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
A number of recent benchmarks seek to assess how well models handle natu...
research
∙
05/23/2023
Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
Large language models (LLMs) have achieved widespread success on a varie...
research
∙
05/07/2023
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
Large Language Models (LLMs) can achieve strong performance on many task...
research
∙
04/02/2023
Eight Things to Know about Large Language Models
The widespread public deployment of large language models (LLMs) in rece...
research
∙
03/28/2023
Improving Code Generation by Training with Natural Language Feedback
The potential for pre-trained large language models (LLMs) to use natura...
research
∙
02/16/2023
Pretraining Language Models with Human Preferences
Language models (LMs) are pretrained to imitate internet text, including...
research
∙
02/15/2023
The Capacity for Moral Self-Correction in Large Language Models
We test the hypothesis that language models trained with reinforcement l...
research
∙
12/20/2022
(QA)^2: Question Answering with Questionable Assumptions
Naturally-occurring information-seeking questions often contain question...
research
∙
12/15/2022
Constitutional AI: Harmlessness from AI Feedback
As AI systems become more capable, we would like to enlist their help to...
research
∙
11/04/2022
Measuring Progress on Scalable Oversight for Large Language Models
Developing safe and useful general-purpose AI systems will require us to...
research
∙
10/19/2022
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions
The use of language-model-based question-answering systems to aid humans...
research
∙
08/26/2022
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
We present the results of the NLP Community Metasurvey. Run from May to ...
research
∙
08/17/2022
What Artificial Neural Networks Can Tell Us About Human Language Acquisition
Rapid progress in machine learning for natural language processing has t...
research
∙
05/23/2022
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
Summarization datasets are often assembled either by scraping naturally ...
research
∙
05/22/2022
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Large language models are able to perform a task by conditioning on a fe...
research
∙
04/11/2022
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
Current QA systems can generate reasonable-sounding yet false answers wi...
research
∙
03/12/2022
What Makes Reading Comprehension Questions Difficult?
For a natural language understanding benchmark to be useful in research,...
research
∙
12/16/2021
QuALITY: Question Answering with Long Input Texts, Yes!
To enable building and testing models on long-document comprehension, we...
research
∙
11/16/2021
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
More capable language models increasingly saturate existing task benchma...
research
∙
10/15/2021
Learning with Noisy Labels by Targeted Relabeling
Crowdsourcing platforms are often used to collect datasets for training ...
research
∙
10/15/2021
When Combating Hype, Proceed with Caution
In an effort to avoid reinforcing widespread hype about the capabilities...
research
∙
10/15/2021
BBQ: A Hand-Built Bias Benchmark for Question Answering
It is well documented that NLP models learn social biases present in the...
research
∙
09/17/2021
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
Despite the success of fine-tuning pretrained language encoders like BER...
research
∙
09/14/2021
NOPE: A Corpus of Naturally-Occurring Presuppositions in English
Understanding language requires grasping not only the overtly stated con...
research
∙
06/01/2021
Comparing Test Sets with Item Response Theory
Recent years have seen numerous NLP datasets introduced to evaluate the ...
research
∙
06/01/2021
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
Crowdsourcing is widely used to create data for common natural language ...
research
∙
04/15/2021
Does Putting a Linguist in the Loop Improve NLU Data Collection?
Many crowdsourced NLP datasets contain systematic gaps and biases that a...
research
∙
04/05/2021
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Evaluation for many natural language understanding (NLU) tasks is broken...
research
∙
11/10/2020
When Do You Need Billions of Words of Pretraining Data?
NLP is currently dominated by general-purpose pretrained language models...
research
∙
10/13/2020
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
Large-scale natural language inference (NLI) datasets such as SNLI or MN...
research
∙
10/11/2020
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
One reason pretraining on self-supervised linguistic tasks is effective ...
research
∙
10/09/2020
Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data
A growing body of work shows that models exploit annotation artifacts to...
research
∙
10/08/2020
Precise Task Formalization Matters in Winograd Schema Evaluations
Performance on the Winograd Schema Challenge (WSC), a respected English ...
research
∙
09/30/2020
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Pretrained language models, especially masked language models (MLMs) hav...
research
∙
07/14/2020
Can neural networks acquire a structural bias from raw linguistic data?
We evaluate whether BERT, a widely used neural network for sentence proc...
research
∙
05/27/2020
Self-Training for Unsupervised Parsing with PRPN
Neural unsupervised parsing (UP) models learn to parse without access to...
research
∙
05/26/2020
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training has been shown to substantially improve pretr...
research
∙
05/01/2020
Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
While pretrained models such as BERT have shown large gains across natur...
research
∙
04/28/2020
Learning to Learn Morphological Inflection for Resource-Poor Languages
We propose to cast the task of morphological inflection - mapping a lemm...
research
∙
04/24/2020
Collecting Entailment Data for Pretraining: New Protocols and Negative Results
Textual entailment (or NLI) data has proven useful as pretraining data f...
research
∙
03/04/2020
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
We introduce jiant, an open source toolkit for conducting multitask and ...
research
∙
12/02/2019
BLiMP: A Benchmark of Linguistic Minimal Pairs for English
We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLi...
research
∙
11/27/2019
Do Attention Heads in BERT Track Syntactic Dependencies?
We investigate the extent to which individual attention heads in pretrai...
research
∙
09/22/2019
Inducing Constituency Trees through Neural Machine Translation
Latent tree learning(LTL) methods learn to parse sentences using only in...
research
∙
09/05/2019
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
Though state-of-the-art sentence representation models can perform tasks...
research
∙
09/04/2019
Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set
Development sets are impractical to obtain for real low-resource languag...
research
∙
05/24/2019
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark
The GLUE benchmark (Wang et al., 2019b) is a suite of language understan...
research
∙
05/24/2019