-
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
Recent work has shown that pre-trained language models such as BERT impr...
read it
-
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
We present CodeBERT, a bimodal pre-trained model for programming languag...
read it
-
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark
The deep learning community has proposed optimizations spanning hardware...
read it
-
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
Large-scale pre-trained multimodal transformers, such as ViLBERT and UNI...
read it
-
Incorporating Count-Based Features into Pre-Trained Models for Improved Stance Detection
The explosive growth and popularity of Social Media has revolutionised t...
read it
-
Unnatural Language Inference
Natural Language Understanding has witnessed a watershed moment with the...
read it
-
What Do Adversarially Robust Models Look At?
In this paper, we address the open question: "What do adversarially robu...
read it
Measuring and Reducing Gendered Correlations in Pre-trained Models
Pre-trained models have revolutionized natural language understanding. However, researchers have found they can encode artifacts undesired in many applications, such as professions correlating with one gender more than another. We explore such gendered correlations as a case study for how to address unintended correlations in pre-trained models. We define metrics and reveal that it is possible for models with similar accuracy to encode correlations at very different rates. We show how measured correlations can be reduced with general-purpose techniques, and highlight the trade offs different strategies have. With these results, we make recommendations for training robust models: (1) carefully evaluate unintended correlations, (2) be mindful of seemingly innocuous configuration differences, and (3) focus on general mitigations.
READ FULL TEXT
Comments
There are no comments yet.