Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

03/20/2023
by   Xinnian Liang, et al.
0

Pretrained language models (PLMs) have shown marvelous improvements across various NLP tasks. Most Chinese PLMs simply treat an input text as a sequence of characters, and completely ignore word information. Although Whole Word Masking can alleviate this, the semantics in words is still not well represented. In this paper, we revisit the segmentation granularity of Chinese PLMs. We propose a mixed-granularity Chinese BERT (MigBERT) by considering both characters and words. To achieve this, we design objective functions for learning both character and word-level representations. We conduct extensive experiments on various Chinese NLP tasks to evaluate existing PLMs as well as the proposed MigBERT. Experimental results show that MigBERT achieves new SOTA performance on all these tasks. Further analysis demonstrates that words are semantically richer than characters. More interestingly, we show that MigBERT also works with Japanese. Our code and model have been released here [https://github.com/xnliang98/MigBERT].

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/23/2023

Retrieval-Augmented Classification with Decoupled Representation

Pretrained language models (PLMs) have shown marvelous improvements acro...
research
04/15/2021

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

Chinese pre-trained language models usually process text as a sequence o...
research
06/01/2021

SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining

Conventional tokenization methods for Chinese pretrained language models...
research
08/18/2023

ChatHaruhi: Reviving Anime Character in Reality via Large Language Model

Role-playing chatbots built on large language models have drawn interest...
research
10/20/2022

Improving Chinese Spelling Check by Character Pronunciation Prediction: The Effects of Adaptivity and Granularity

Chinese spelling check (CSC) is a fundamental NLP task that detects and ...
research
03/14/2023

Good Neighbors Are All You Need for Chinese Grapheme-to-Phoneme Conversion

Most Chinese Grapheme-to-Phoneme (G2P) systems employ a three-stage fram...
research
08/28/2019

Onto Word Segmentation of the Complete Tang Poems

We aim at segmenting words in the Complete Tang Poems (CTP). Although it...

Please sign up or login with your details

Forgot password? Click here to reset