DeepAI AI Chat
Log In Sign Up

RoBERTa-wwm-ext Fine-Tuning for Chinese Text Classification

by   Zhuo Xu, et al.

Bidirectional Encoder Representations from Transformers (BERT) have shown to be a promising way to dramatically improve the performance across various Natural Language Processing tasks [Devlin et al., 2019]. Meanwhile, progress made over the past few years by various Neural Net-work has also proved the effectiveness of Neural Network in the field of Natural Language Processing. In this project, RoBERTa-wwm-ext [Cui et al., 2019] pre-train language model was adopted and fine-tuned for Chinese text classification. The models were able to classify Chinese texts into two categories, containing descriptions of legal behavior and descriptions of illegal behavior. Four different models are also proposed in the paper. Those models will use RoBERTa-wwm-extas their embedding layer and feed the embedding into different neural networks. The motivation be-hind proposing these models is straightforward. By introducing complex output layer architecture, the overall performance of the models could be improved. All the models were trained on a data set derived from Chinese public court records, and the performance of different models were compared.The experiment shows that the performance of pro-posed models failed to beat the original RoBERTa-wwm-ext model in terms of accuracy and training efficiency.


page 1

page 2

page 3

page 4


Ad Text Classification with Transformer-Based Natural Language Processing Methods

In this study, a natural language processing-based (NLP-based) method is...

BERT-based Chinese Text Classification for Emergency Domain with a Novel Loss Function

This paper proposes an automatic Chinese text categorization method for ...

Recognition and Processing of NATOM

In this paper we show how to process the NOTAM (Notice to Airmen) data o...

What the [MASK]? Making Sense of Language-Specific BERT Models

Recently, Natural Language Processing (NLP) has witnessed an impressive ...

Profitable Trade-Off Between Memory and Performance In Multi-Domain Chatbot Architectures

Text classification problem is a very broad field of study in the field ...

Stacked DeBERT: All Attention in Incomplete Data for Text Classification

In this paper, we propose Stacked DeBERT, short for Stacked Denoising Bi...

The relational processing limits of classic and contemporary neural network models of language processing

The ability of neural networks to capture relational knowledge is a matt...