Language Detoxification with Attribute-Discriminative Latent Space

10/19/2022
by   Jin Myung Kwak, et al.
3

Transformer-based Language Models (LMs) achieve remarkable performances on a variety of NLU tasks, but are also prone to generating toxic texts such as insults, threats, and profanities which limit their adaptations to the real-world applications. To overcome this issue, a few text generation approaches aim to detoxify toxic texts with additional LMs or perturbations. However, previous methods require excessive memory, computations, and time which are serious bottlenecks in their real-world application. To address such limitations, we propose an effective yet efficient method for language detoxification using an attribute-discriminative latent space. Specifically, we project the latent space of an original Transformer LM to a discriminative latent space on which the texts are well-separated by their attributes, with the help of a projection block and a discriminator. This allows the LM to control the text generation to be non-toxic with minimal memory and computation overhead. We validate our model, Attribute-Discriminative Language Model (ADLM) on detoxified language and dialogue generation tasks, on which our method significantly outperforms baselines both in performance and efficiency.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/20/2021

Attribute Alignment: Controlling Text Generation from Pre-trained Language Models

Large language models benefit from training with a large amount of unlab...
research
12/16/2022

Controllable Text Generation via Probability Density Estimation in the Latent Space

Previous work on controllable text generation has explored the idea of c...
research
07/26/2019

Latent Space Factorisation and Manipulation via Matrix Subspace Projection

This paper proposes a novel method for factorising the information in th...
research
08/01/2022

Composable Text Control Operations in Latent Space with Ordinary Differential Equations

Real-world text applications often involve composing a wide range of tex...
research
10/18/2022

DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation

Prompt learning with immensely large Casual Language Models (CLMs) has b...
research
10/16/2022

Model Criticism for Long-Form Text Generation

Language models have demonstrated the ability to generate highly fluent ...
research
05/22/2023

MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space

Multi-aspect controllable text generation aims to generate fluent senten...

Please sign up or login with your details

Forgot password? Click here to reset