SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

09/21/2020
by   Nathan Ng, et al.
15

Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8 1.8

READ FULL TEXT

page 1

page 2

page 3

page 4

04/24/2020

G-DAUG: Generative Data Augmentation for Commonsense Reasoning

Recent advances in commonsense reasoning depend on large-scale human-ann...
07/05/2022

Predicting Out-of-Domain Generalization with Local Manifold Smoothness

Understanding how machine learning models generalize to new environments...
04/25/2022

VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalization

Invariance to diverse types of image corruption, such as noise, blurring...
04/02/2021

Data Augmentation with Manifold Barycenters

The training of Generative Adversarial Networks (GANs) requires a large ...
02/25/2022

PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks

This paper focuses on the Data Augmentation for low-resource Natural Lan...
08/13/2021

FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning

Most previous methods for text data augmentation are limited to simple t...
02/22/2021

MixUp Training Leads to Reduced Overfitting and Improved Calibration for the Transformer Architecture

MixUp is a computer vision data augmentation technique that uses convex ...

Code Repositories