research
∙
08/31/2023
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
We present Belebele, a multiple-choice machine reading comprehension (MR...
share
research
∙
08/31/2023
The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages
Gender biases in language generation systems are challenging to mitigate...
research
∙
08/11/2023
Self-Alignment with Instruction Backtranslation
We present a scalable method to build a high quality instruction followi...
research
∙
08/08/2023
Shepherd: A Critic for Language Model Generation
As large language models improve, there is increasing interest in techni...
research
∙
08/08/2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
The legality of training language models (LMs) on copyrighted or otherwi...
research
∙
07/31/2023
Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?
Coaxing out desired behavior from pretrained models, while avoiding unde...
research
∙
05/24/2023
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Language models (LMs) often struggle to pay enough attention to the inpu...
research
∙
05/24/2023
Mixture of Prompt Experts for Generalizable and Interpretable Question Answering
One of the ultimate quests of question answering (QA) is to deploy a sys...
research
∙
05/23/2023
QLoRA: Efficient Finetuning of Quantized LLMs
We present QLoRA, an efficient finetuning approach that reduces memory u...
research
∙
05/23/2023
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Evaluating the factuality of long-form text generated by large language ...
research
∙
05/23/2023
Revisiting Machine Translation for Cross-lingual Classification
Machine Translation (MT) has been widely used for cross-lingual classifi...
research
∙
05/18/2023
LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretra...
research
∙
05/12/2023
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Autoregressive transformers are spectacular models for short sequences b...
research
∙
04/26/2023
Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models
Pretrained Language Models (PLMs) learn rich cross-lingual knowledge and...
research
∙
04/25/2023
Stable and low-precision training for large-scale vision-language models
We introduce new methods for 1) accelerating and 2) stabilizing training...
research
∙
03/24/2023
Scaling Expert Language Models with Unsupervised Domain Discovery
Large language models are typically trained densely: all parameters are ...
research
∙
03/16/2023
ART: Automatic multi-step reasoning and tool-use for large language models
Large language models (LLMs) can perform complex reasoning in few- and z...
research
∙
02/15/2023
Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation
Large language models (LLMs) demonstrate remarkable machine translation ...
research
∙
02/09/2023
Toolformer: Language Models Can Teach Themselves to Use Tools
Language models (LMs) exhibit remarkable abilities to solve new tasks fr...
research
∙
02/04/2023
Representation Deficiency in Masked Language Modeling
Masked Language Modeling (MLM) has been one of the most prominent approa...
research
∙
01/30/2023
REPLUG: Retrieval-Augmented Black-Box Language Models
We introduce REPLUG, a retrieval-augmented language modeling framework t...
research
∙
01/25/2023
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Large multilingual language models typically rely on a single vocabulary...
research
∙
01/10/2023
Scaling Laws for Generative Mixed-Modal Language Models
Generative language models define distributions over sequences of tokens...
research
∙
01/05/2023
CiT: Curation in Training for Effective Vision-Language Data
Large vision-language models are generally applicable to many downstream...
research
∙
12/22/2022
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Recent work has shown that fine-tuning large pre-trained language models...
research
∙
12/20/2022
Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?
Large language models can perform new tasks in a zero-shot fashion, give...
research
∙
12/19/2022
Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations
Although large language models can be prompted for both zero- and few-sh...
research
∙
12/19/2022
Training Trajectories of Language Models Across Scales
Scaling up language models has led to unprecedented performance gains, b...
research
∙
12/19/2022
The case for 4-bit precision: k-bit Inference Scaling Laws
Quantization methods reduce the number of bits required to represent eac...
research
∙
12/15/2022
ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
Large language models show improved downstream task performance when pro...
research
∙
12/08/2022
Demystifying Prompts in Language Models via Perplexity Estimation
Language models can be prompted to perform a wide variety of zero- and f...
research
∙
12/05/2022
In-context Examples Selection for Machine Translation
Large-scale generative models show an impressive ability to perform a wi...
research
∙
12/02/2022
Nonparametric Masked Language Modeling
Existing language models (LMs) predict tokens with a softmax over a fini...
research
∙
12/02/2022
AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization
Models trained via empirical risk minimization (ERM) are known to rely o...
research
∙
11/30/2022
CREPE: Open-Domain Question Answering with False Presuppositions
Information seeking users often pose questions with false presupposition...
research
∙
11/22/2022
Retrieval-Augmented Multimodal Language Modeling
Recent multimodal models such as DALL-E and CM3 have achieved remarkable...
research
∙
11/15/2022
Prompting Language Models for Linguistic Structure
Although pretrained language models (PLMs) can be prompted to perform a ...
research
∙
10/27/2022
Contrastive Decoding: Open-ended Text Generation as Optimization
Likelihood, although useful as a training loss, is a poor search objecti...
research
∙
10/25/2022
RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering
We introduce RoMQA, the first benchmark for robust, multi-evidence, mult...
research
∙
10/13/2022
M2D2: A Massively Multi-domain Language Modeling Dataset
We present M2D2, a fine-grained, massively multi-domain corpus for study...
research
∙
10/10/2022
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Counterfactual data augmentation (CDA) – i.e., adding minimally perturbe...
research
∙
10/06/2022
Binding Language Models in Symbolic Languages
Though end-to-end neural approaches have recently been dominating NLP ta...
research
∙
09/30/2022
Improving Policy Learning via Language Dynamics Distillation
Recent work has shown that augmenting environments with language descrip...
research
∙
09/21/2022
Mega: Moving Average Equipped Gated Attention
The design choices in the Transformer attention mechanism, including wea...
research
∙
08/15/2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Large language models have been widely adopted but require significant G...
research
∙
08/05/2022
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
We present Branch-Train-Merge (BTM), a communication-efficient algorithm...
research
∙
06/21/2022
Questions Are All You Need to Train a Dense Passage Retriever
We introduce ART, a new corpus-level autoencoding approach for training ...
research
∙
06/07/2022
LegoNN: Building Modular Encoder-Decoder Models
State-of-the-art encoder-decoder models (e.g. for machine translation (M...
research
∙
05/27/2022
Nearest Neighbor Zero-Shot Inference
We introduce kNN-Prompt, a simple and effective technique to use k-neare...
research
∙
05/25/2022