RoBERTa: A Robustly Optimized BERT Pretraining Approach

by   Yinhan Liu, et al.
University of Washington

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.


page 1

page 2

page 3

page 4


How to Train BERT with an Academic Budget

While large language models à la BERT are used ubiquitously in NLP, pret...

Pretraining Without Attention

Transformers have been essential to pretraining success in NLP. Other ar...

Cloze-driven Pretraining of Self-attention Networks

We present a new approach for pretraining a bi-directional transformer m...

HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

BERT-based models are currently used for solving nearly all Natural Lang...

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Pretraining NLP models with variants of Masked Language Model (MLM) obje...

ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understanding

We present ToddlerBERTa, a BabyBERTa-like language model, exploring its ...

The MultiBERTs: BERT Reproductions for Robustness Analysis

Experiments with pretrained models such as BERT are often based on a sin...

Please sign up or login with your details

Forgot password? Click here to reset