BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation

04/16/2022
by   Zheng Zhang, et al.
0

Data augmentations (DA) are the cores to achieving robust sequence-to-sequence learning on various natural language processing (NLP) tasks. However, most of the DA approaches force the decoder to make predictions conditioned on the perturbed input representation, underutilizing supervised information provided by perturbed input. In this work, we propose a framework-level robust sequence-to-sequence learning approach, named BLISS, via self-supervised input representation, which has the great potential to complement the data-level augmentation approaches. The key idea is to supervise the sequence-to-sequence framework with both the supervised ("input→output") and self-supervised ("perturbed input→input") information. We conduct comprehensive experiments to validate the effectiveness of BLISS on various tasks, including machine translation, grammatical error correction, and text summarization. The results show that BLISS outperforms significantly the vanilla Transformer and consistently works well across tasks than the other five contrastive baselines. Extensive analyses reveal that BLISS learns robust representations and rich linguistic knowledge, confirming our claim. Source code will be released upon publication.

READ FULL TEXT
research
12/29/2020

Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning

Encoder layer fusion (EncoderFusion) is a technique to fuse all the enco...
research
11/01/2022

Self-supervised Character-to-Character Distillation

Handling complicated text images (e.g., irregular structures, low resolu...
research
02/07/2022

Graph Self-supervised Learning with Accurate Discrepancy Learning

Self-supervised learning of graph neural networks (GNNs) aims to learn a...
research
05/30/2022

E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation

Sequence-to-sequence (seq2seq) learning has become a popular trend for p...
research
09/15/2021

SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations

While contrastive learning is proven to be an effective training strateg...
research
08/26/2021

Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization

In this paper, we present a denoising sequence-to-sequence (seq2seq) aut...
research
05/20/2019

A Neural, Interactive-predictive System for Multimodal Sequence to Sequence Tasks

We present a demonstration of a neural interactive-predictive system for...

Please sign up or login with your details

Forgot password? Click here to reset