research
∙
04/21/2023
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
It is widely acknowledged that large models have the potential to delive...
share
research
∙
05/02/2022
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands...
research
∙
03/14/2022
Efficient Language Modeling with Sparse all-MLP
All-MLP architectures have attracted increasing interest as an alternati...
research
∙
12/20/2021
Efficient Large Scale Language Modeling with Mixtures of Experts
Mixture of Experts layers (MoEs) enable efficient scaling of language mo...
research
∙
12/20/2021
Few-shot Learning with Multilingual Language Models
Large-scale autoregressive language models such as GPT-3 are few-shot le...
research
∙
10/30/2021
Sustainable AI: Environmental Implications, Challenges and Opportunities
This paper explores the environmental impact of the super-linear growth ...
research
∙
10/18/2021
NormFormer: Improved Transformer Pretraining with Extra Normalization
During pretraining, the Pre-LayerNorm transformer suffers from a gradien...
research
∙
06/17/2021
On Anytime Learning at Macroscale
Classical machine learning frameworks assume access to a possibly large ...
research
∙
05/02/2021
Larger-Scale Transformers for Multilingual Masked Language Modeling
Recent work has demonstrated the effectiveness of cross-lingual language...
research
∙
12/17/2020
Few-shot Sequence Learning with Transformers
Few-shot algorithms aim at learning new tasks provided only a handful of...
research
∙
04/29/2020
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
The state of the art on many NLP tasks is currently achieved by large pr...
research
∙
04/28/2020
Recipes for building an open-domain chatbot
Building open-domain chatbots is a challenging area for machine learning...
research
∙
04/22/2020
Residual Energy-Based Models for Text Generation
Text generation is ubiquitous in many NLP tasks, from summarization, to ...
research
∙
11/09/2019
How Decoding Strategies Affect the Verifiability of Generated Text
Language models are of considerable importance. They are used for pretra...
research
∙
11/05/2019
Unsupervised Cross-lingual Representation Learning at Scale
This paper shows that pretraining multilingual language models at scale ...
research
∙
10/16/2019
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models
In this work, we study how the large-scale pretrain-finetune framework c...
research
∙
10/15/2019
Facebook AI's WAT19 Myanmar-English Translation Task Submission
This paper describes Facebook AI's submission to the WAT 2019 Myanmar-En...
research
∙
09/28/2019
The Source-Target Domain Mismatch Problem in Machine Translation
While we live in an increasingly interconnected world, different places ...
research
∙
08/14/2019
On The Evaluation of Machine Translation Systems Trained With Back-Translation
Back-translation is a widely used data augmentation technique which leve...
research
∙
07/26/2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but ...
research
∙
07/15/2019
Facebook FAIR's WMT19 News Translation Task Submission
This paper describes Facebook FAIR's submission to the WMT19 shared news...
research
∙
06/07/2019
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Recent advances in generative modeling of text have demonstrated remarka...
research
∙
04/01/2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
fairseq is an open-source sequence modeling toolkit that allows research...
research
∙
02/20/2019
Mixture Models for Diverse Machine Translation: Tricks of the Trade
Mixture models trained via EM are among the simplest, most widely used a...
research
∙
02/04/2019
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English
The vast majority of language pairs in the world are low-resource becaus...
research
∙
08/28/2018
Understanding Back-Translation at Scale
An effective method to improve neural machine translation with monolingu...
research
∙
06/01/2018
Scaling Neural Machine Translation
Sequence to sequence learning models still require several days to reach...
research
∙
04/20/2018
Phrase-Based & Neural Unsupervised Machine Translation
Machine translation systems achieve near human-level performance on some...
research
∙
02/28/2018
Analyzing Uncertainty in Neural Machine Translation
Machine translation is a popular test bed for research in neural sequenc...
research
∙
11/14/2017