CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

by   Yunfan Shao, et al.

In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed for both natural language understanding (NLU) and natural language generation (NLG) tasks. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced Transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.


page 1

page 2

page 3

page 4


Auto-Learning: An Adversarial Process of Two Pre-trained Models for Natural Language Generation

Pre-trained models have been used in many fields in recent years, rangin...

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

When trained effectively, the Variational Autoencoder (VAE) can be both ...

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator

Pre-trained models have achieved remarkable success in natural language ...

INSET: Sentence Infilling with Inter-sentential Generative Pre-training

Missing sentence generation (or sentence infilling) fosters a wide range...

TransCouplet:Transformer based Chinese Couplet Generation

Chinese couplet is a special form of poetry composed of complex syntax w...

RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation

Can AI help automate human-easy but computer-hard data preparation tasks...

ChatGPT: Applications, Opportunities, and Threats

Developed by OpenAI, ChatGPT (Conditional Generative Pre-trained Transfo...

Code Repositories


CPT: A Pre-Trained Unbalanced Transformerfor Both Chinese Language Understanding and Generation

view repo

Please sign up or login with your details

Forgot password? Click here to reset