research
∙
08/27/2023
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
The ability of large language models (LLMs) to follow natural language i...
share
research
∙
07/28/2023
Robust Distortion-free Watermarks for Language Models
We propose a methodology for planting watermarks in text from an autoreg...
research
∙
07/12/2023
Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
Machine learning is traditionally studied at the model level: researcher...
research
∙
07/06/2023
Lost in the Middle: How Language Models Use Long Contexts
While recent language models have the ability to take long contexts as i...
research
∙
06/16/2023
Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness
Language model training in distributed settings is limited by the commun...
research
∙
06/14/2023
Anticipatory Music Transformer
We introduce anticipation: a method for constructing a controllable gene...
research
∙
06/05/2023
Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment
We present the NeurIPS 2021 consistency experiment, a larger-scale varia...
research
∙
05/27/2023
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models
Language models have been shown to exhibit positive scaling, where perfo...
research
∙
05/24/2023
Lexinvariant Language Models
Token embeddings, a mapping from discrete lexical symbols to continuous ...
research
∙
05/23/2023
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
Given the massive cost of language model pre-training, a non-trivial imp...
research
∙
05/21/2023
PRODIGY: Enabling In-context Learning Over Graphs
In-context learning is the ability of a pretrained model to adapt to nov...
research
∙
05/17/2023
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e.g., Wikipedia, bo...
research
∙
05/03/2023
Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs
Large language models (LLMs) power many state-of-the-art systems in natu...
research
∙
04/19/2023
Evaluating Verifiability in Generative Search Engines
Generative search engines directly generate responses to user queries, a...
research
∙
04/07/2023
Generative Agents: Interactive Simulacra of Human Behavior
Believable proxies of human behavior can empower interactive application...
research
∙
03/30/2023
Whose Opinions Do Language Models Reflect?
Language models (LMs) are increasingly being used in open-ended contexts...
research
∙
03/28/2023
Ecosystem Graphs: The Social Footprint of Foundation Models
Foundation models (e.g. ChatGPT, StableDiffusion) pervasively influence ...
research
∙
03/28/2023
Foundation Models and Fair Use
Existing foundation models are trained on copyrighted material. Deployin...
research
∙
03/13/2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
The high computational and memory requirements of large language model (...
research
∙
02/24/2023
Language-Driven Representation Learning for Robotics
Recent work in visual representation learning for robotics demonstrates ...
research
∙
02/23/2023
Out-of-Domain Robustness via Targeted Augmentations
Models trained on one set of domains often suffer performance drops on u...
research
∙
02/06/2023
Data Selection for Language Models via Importance Resampling
Selecting a suitable training dataset is crucial for both general-domain...
research
∙
01/31/2023
Benchmarking Large Language Models for News Summarization
Large language models (LLMs) have shown promise for automatic summarizat...
research
∙
01/06/2023
"No, to the Right" – Online Language Corrections for Robotic Manipulation via Shared Autonomy
Systems for language-guided human-robot interaction must satisfy two key...
research
∙
12/28/2022
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Retrieval-augmented in-context learning has emerged as a powerful approa...
research
∙
12/20/2022
Trustworthy Social Bias Measurement
How do we design measures of social bias that we trust? While prior work...
research
∙
12/19/2022
Evaluating Human-Language Model Interaction
Many real-world applications of language models (LMs), such as code auto...
research
∙
12/04/2022
Melody transcription via generative pre-training
Despite the central role that melody plays in music perception, it remai...
research
∙
11/25/2022
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme ...
research
∙
11/22/2022
Retrieval-Augmented Multimodal Language Modeling
Recent multimodal models such as DALL-E and CM3 have achieved remarkable...
research
∙
11/22/2022
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
How do author perceptions match up to the outcomes of the peer-review pr...
research
∙
11/16/2022
Holistic Evaluation of Language Models
Language models (LMs) are becoming the foundation for almost all major l...
research
∙
10/27/2022
Truncation Sampling as Language Model Desmoothing
Long samples of text from neural language models can be of poor quality....
research
∙
10/27/2022
Contrastive Decoding: Open-ended Text Generation as Optimization
Likelihood, although useful as a training loss, is a poor search objecti...
research
∙
10/20/2022
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
A common approach to transfer learning under distribution shift is to fi...
research
∙
10/17/2022
Deep Bidirectional Language-Knowledge Graph Pretraining
Pretraining a language model (LM) on text has been shown to help various...
research
∙
10/12/2022
Are Sample-Efficient NLP Models More Robust?
Recent work has observed that pre-trained models have higher out-of-dist...
research
∙
08/08/2022
Social Simulacra: Creating Populated Prototypes for Social Computing Systems
Social computing prototypes probe the social behaviors that may arise in...
research
∙
08/01/2022
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
In-context learning refers to the ability of a model to condition on a p...
research
∙
07/18/2022
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift
We often see undesirable tradeoffs in robust machine learning where out-...
research
∙
06/21/2022
Insights into Pre-training via Simpler Synthetic Tasks
Pre-training produces representations that are effective for a wide rang...
research
∙
06/02/2022
Decentralized Training of Foundation Models in Heterogeneous Environments
Training foundation models, such as GPT-3 and PaLM, can be extremely exp...
research
∙
05/27/2022
Diffusion-LM Improves Controllable Text Generation
Controlling the behavior of language models (LMs) without re-training is...
research
∙
04/01/2022
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
We consider unsupervised domain adaptation (UDA), where labeled data fro...
research
∙
03/29/2022
LinkBERT: Pretraining Language Models with Document Links
Language model (LM) pretraining can learn various knowledge from text co...
research
∙
02/21/2022
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
When transferring a pretrained model to a downstream task, two popular m...
research
∙
01/21/2022
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Answering complex questions about textual narratives requires reasoning ...
research
∙
01/18/2022
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
Large language models (LMs) offer unprecedented language generation capa...
research
∙
12/09/2021
Extending the WILDS Benchmark for Unsupervised Adaptation
Machine learning systems deployed in the wild are often trained on a sou...
research
∙
11/05/2021