Using Pre-Training Can Improve Model Robustness and Uncertainty

by   Dan Hendrycks, et al.

Tuning a pre-trained network is commonly thought to improve data efficiency. However, Kaiming He et al. have called into question the utility of pre-training by showing that training from scratch can often yield similar performance, should the model train long enough. We show that although pre-training may not improve performance on traditional classification metrics, it does provide large benefits to model robustness and uncertainty. Through extensive experiments on label corruption, class imbalance, adversarial examples, out-of-distribution detection, and confidence calibration, we demonstrate large gains from pre-training and complementary effects with task-specific methods. We show approximately a 30 label noise robustness and a 10 on CIFAR-10 and CIFAR-100. In some cases, using pre-training without task-specific methods surpasses the state-of-the-art, highlighting the importance of using pre-training when evaluating future methods on robustness and uncertainty tasks.


Adversarial Training for Large Neural Language Models

Generalization and robustness are both key desiderata for designing mach...

On the Connection between Pre-training Data Diversity and Fine-tuning Robustness

Pre-training has been widely adopted in deep learning to improve model p...

Evaluating Synthetic Pre-Training for Handwriting Processing Tasks

In this work, we explore massive pre-training on synthetic word images f...

Predictions For Pre-training Language Models

Language model pre-training has proven to be useful in many language und...

Does Optimal Source Task Performance Imply Optimal Pre-training for a Target Task?

Pre-trained deep nets are commonly used to improve accuracies and traini...

Ontology Pre-training for Poison Prediction

Integrating human knowledge into neural networks has the potential to im...

TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training

We present the TAPAS contribution to the Shared Task on Statement Verifi...

Code Repositories


Pre-Training Buys Better Robustness and Uncertainty Estimates (ICML 2019)

view repo

Please sign up or login with your details

Forgot password? Click here to reset