research
∙
08/08/2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
The legality of training language models (LMs) on copyrighted or otherwi...
share
research
∙
06/05/2023
Information Flow Control in Machine Learning through Modular Model Architecture
In today's machine learning (ML) models, any part of the training data c...
research
∙
03/24/2023
Scaling Expert Language Models with Unsupervised Domain Discovery
Large language models are typically trained densely: all parameters are ...
research
∙
12/08/2022
Editing Models with Task Arithmetic
Changing how pre-trained models behave – e.g., improving their performan...
research
∙
10/19/2022
lo-fi: distributed fine-tuning without communication
When fine-tuning large neural networks, it is common to use multiple nod...
research
∙
10/13/2022
M2D2: A Massively Multi-domain Language Modeling Dataset
We present M2D2, a fine-grained, massively multi-domain corpus for study...
research
∙
08/05/2022
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
We present Branch-Train-Merge (BTM), a communication-efficient algorithm...
research
∙
05/27/2022
Nearest Neighbor Zero-Shot Inference
We introduce kNN-Prompt, a simple and effective technique to use k-neare...
research
∙
01/25/2022
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
Language models increasingly rely on massive web dumps for diverse text ...
research
∙
11/14/2021
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
When an NLP model is trained on text data from one time period and teste...
research
∙
10/01/2021
Expected Validation Performance and Estimation of a Random Variable's Maximum
Research in NLP is often supported by experimental results, and improved...
research
∙
08/11/2021
DEMix Layers: Disentangling Domains for Modular Language Modeling
We introduce a new domain expert mixture (DEMix) layer that enables cond...
research
∙
06/30/2021
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Human evaluations are typically considered the gold standard in natural ...
research
∙
04/13/2021
Detoxifying Language Models Risks Marginalizing Minority Voices
Language models (LMs) must be both safe and equitable to be responsibly ...
research
∙
09/24/2020
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Pretrained neural language models (LMs) are prone to generating racist, ...
research
∙
04/23/2020
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Language models pretrained on text from a wide variety of sources form t...
research
∙
09/06/2019
Show Your Work: Improved Reporting of Experimental Results
Research in natural language processing proceeds, in part, by demonstrat...
research
∙
06/05/2019
Variational Pretraining for Semi-supervised Text Classification
We introduce VAMPIRE, a lightweight pretraining framework for effective ...
research
∙
03/06/2018