Luke Zettlemoyer

research

∙ 06/21/2022

Questions Are All You Need to Train a Dense Passage Retriever

We introduce ART, a new corpus-level autoencoding approach for training ...

Devendra Singh Sachan, et al. ∙

research

∙ 06/07/2022

LegoNN: Building Modular Encoder-Decoder Models

State-of-the-art encoder-decoder models (e.g. for machine translation (M...

Siddharth Dalmia, et al. ∙

research

∙ 05/27/2022

Nearest Neighbor Zero-Shot Inference

We introduce kNN-Prompt, a simple and effective technique to use k-neare...

Weijia Shi, et al. ∙

research

∙ 05/25/2022

Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI

Evaluating an explanation's faithfulness is desired for many reasons suc...

Suzanna Sia, et al. ∙

research

∙ 05/24/2022

Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

The emergent cross-lingual transfer seen in multilingual pretrained mode...

Terra Blevins, et al. ∙

research

∙ 05/24/2022

On the Role of Bidirectionality in Language Model Pre-Training

Prior work on language model pre-training has explored different archite...

Mikel Artetxe, et al. ∙

research

∙ 05/22/2022

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Despite their wide adoption, the underlying training and memorization dy...

Kushal Tirumala, et al. ∙

research

∙ 05/09/2022

Few-shot Mining of Naturally Occurring Inputs and Outputs

Creating labeled natural language training data is expensive and require...

Mandar Joshi, et al. ∙

research

∙ 05/02/2022

OPT: Open Pre-trained Transformer Language Models

Large language models, which are often trained for hundreds of thousands...

Susan Zhang, et al. ∙

research

∙ 04/25/2022

Natural Language to Code Translation with Execution

Generative models of code, pretrained on large corpora of programs, have...

Freda Shi, et al. ∙

research

∙ 04/17/2022

Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models

English pretrained language models, which make up the backbone of many m...

Terra Blevins, et al. ∙

research

∙ 04/15/2022

Improving Passage Retrieval with Zero-Shot Question Generation

We propose a simple and effective re-ranking method for improving passag...

Devendra Singh Sachan, et al. ∙

research

∙ 04/12/2022

InCoder: A Generative Model for Code Infilling and Synthesis

Code is seldom written in a single left-to-right pass and is instead rep...

Daniel Fried, et al. ∙

research

∙ 04/03/2022

PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models

Current methods for few-shot fine-tuning of pretrained masked language m...

Rabeeh Karimi Mahabadi, et al. ∙

research

∙ 02/25/2022

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Large language models (LMs) are able to in-context learn – perform a new...

Sewon Min, et al. ∙

research

∙ 01/25/2022

Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection

Language models increasingly rely on massive web dumps for diverse text ...

Suchin Gururangan, et al. ∙

research

∙ 01/19/2022

CM3: A Causal Masked Multimodal Model of the Internet

We introduce CM3, a family of causally masked generative models trained ...

Armen Aghajanyan, et al. ∙

research

∙ 12/20/2021

Efficient Large Scale Language Modeling with Mixtures of Experts

Mixture of Experts layers (MoEs) enable efficient scaling of language mo...

Mikel Artetxe, et al. ∙

research

∙ 12/20/2021

Few-shot Learning with Multilingual Language Models

Large-scale autoregressive language models such as GPT-3 are few-shot le...

Xi Victoria Lin, et al. ∙

research

∙ 12/07/2021

Reducing Target Group Bias in Hate Speech Detectors

The ubiquity of offensive and hateful content on online fora necessitate...

Darsh J Shah, et al. ∙

research

∙ 12/06/2021

Quantifying Adaptability in Pre-trained Language Models with 500 Tasks

When a neural language model (LM) is adapted to perform a new task, what...

Belinda Z. Li, et al. ∙

research

∙ 11/12/2021

BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation

Mined bitexts can contain imperfect translations that yield unreliable t...

Eleftheria Briakou, et al. ∙

research

∙ 10/29/2021

MetaICL: Learning to Learn In Context

We introduce MetaICL (Meta-training for In-Context Learning), a new meta...

Sewon Min, et al. ∙

research

∙ 10/20/2021

SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark

Existing work in language grounding typically study single environments....

Victor Zhong, et al. ∙

research

∙ 10/06/2021

8-bit Optimizers via Block-wise Quantization

Stateful optimizers maintain gradient statistics over time, e.g., the ex...

Tim Dettmers, et al. ∙

research

∙ 09/28/2021

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

We present VideoCLIP, a contrastive approach to pre-train a unified mode...

Hu Xu, et al. ∙

research

∙ 08/11/2021

DEMix Layers: Disentangling Domains for Modular Language Modeling

We introduce a new domain expert mixture (DEMix) layer that enables cond...

Suchin Gururangan, et al. ∙

research

∙ 08/09/2021

Noisy Channel Language Model Prompting for Few-Shot Text Classification

We introduce a noisy channel approach for language model prompting in fe...

Sewon Min, et al. ∙

research

∙ 07/26/2021

Language Grounding with 3D Objects

Seemingly simple natural language requests to a robot are generally unde...

Jesse Thomason, et al. ∙

research

∙ 07/14/2021

HTLM: Hyper-Text Pre-Training and Prompting of Language Models

We introduce HTLM, a hyper-text language model trained on a large-scale ...

Armen Aghajanyan, et al. ∙

research

∙ 07/05/2021

FaVIQ: FAct Verification from Information-seeking Questions

Despite significant interest in developing general purpose fact checking...

Jungsoo Park, et al. ∙

research

∙ 06/15/2021

Question Answering Infused Pre-training of General-Purpose Contextualized Representations

This paper proposes a pre-training objective based on question answering...

Robin Jia, et al. ∙

research

∙ 06/12/2021

Prompting Contrastive Explanations for Commonsense Reasoning Tasks

Many commonsense reasoning NLP tasks involve choosing between one or mor...

Bhargavi Paranjape, et al. ∙

research

∙ 06/09/2021

DESCGEN: A Distantly Supervised Dataset for Generating Abstractive Entity Descriptions

Short textual descriptions of entities provide summaries of their key at...

Weijia Shi, et al. ∙

research

∙ 06/03/2021

Luna: Linear Unified Nested Attention

The quadratic computational and memory complexities of the Transformer's...

Xuezhe Ma, et al. ∙

research

∙ 05/20/2021

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

We present a simplified, task-agnostic multi-modal pre-training approach...

Hu Xu, et al. ∙

research

∙ 04/16/2021

Surface Form Competition: Why the Highest Probability Answer Isn't Always Right

Large language models have shown promising results in zero-shot settings...

Ari Holtzman, et al. ∙

research

∙ 03/30/2021

BASE Layers: Simplifying Training of Large, Sparse Models

We introduce a new balanced assignment of experts (BASE) layer for large...

Mike Lewis, et al. ∙

research

∙ 03/23/2021

Multilingual Autoregressive Entity Linking

We present mGENRE, a sequence-to-sequence system for the Multilingual En...

Nicola De Cao, et al. ∙

research

∙ 02/16/2021

FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary

Current models for Word Sense Disambiguation (WSD) struggle to disambigu...

Terra Blevins, et al. ∙

research

∙ 01/26/2021

Muppet: Massive Multi-task Representations with Pre-Finetuning

We propose pre-finetuning, an additional large-scale learning stage betw...

Armen Aghajanyan, et al. ∙

research

∙ 01/01/2021

Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment

Bilingual lexicons map words in one language to their translations in an...

Haoyue Shi, et al. ∙

research

∙ 12/22/2020

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Although pretrained language models can be fine-tuned to produce state-o...

Armen Aghajanyan, et al. ∙

research

∙ 11/07/2020

Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles

Many datasets have been shown to contain incidental correlations created...

Christopher Clark, et al. ∙

research

∙ 11/05/2020

Detecting Hallucinated Content in Conditional Neural Sequence Generation

Neural sequence models can generate highly fluent sentences but recent s...

Chunting Zhou, et al. ∙

research

∙ 10/07/2020

Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing

Task-oriented semantic parsing is a critical component of virtual assist...

Xilun Chen, et al. ∙

research

∙ 10/01/2020

Nearest Neighbor Machine Translation

We introduce k-nearest-neighbor machine translation (kNN-MT), which pred...

Urvashi Khandelwal, et al. ∙

research

∙ 09/16/2020

Grounded Adaptation for Zero-shot Executable Semantic Parsing

We propose Grounded Adaptation for Zero-shot Executable Semantic Parsing...

Victor Zhong, et al. ∙

research

∙ 08/06/2020

Better Fine-Tuning by Reducing Representational Collapse

Although widely adopted, existing approaches for fine-tuning pre-trained...

Armen Aghajanyan, et al. ∙

research

∙ 08/03/2020

DeLighT: Very Deep and Light-weight Transformer

We introduce a very deep and light-weight transformer, DeLighT, that del...

Sachin Mehta, et al. ∙

Luke Zettlemoyer

Featured Co-authors