Onto Word Segmentation of the Complete Tang Poems

by   Chao-Lin Liu, et al.

We aim at segmenting words in the Complete Tang Poems (CTP). Although it is possible to do some research about CTP without doing full-scale word segmentation, we must move forward to word-level analysis of CTP for conducting advanced research topics. In November 2018 when we submitted the manuscript for DH 2019 (ADHO), we collected only 2433 poems that were segmented by trained experts, and used the segmented poems to evaluate the segmenter that considered domain knowledge of Chinese poetry. We trained pointwise mutual information (PMI) between Chinese characters based on the CTP poems (excluding the 2433 poems, which were used exclusively only for testing) and the domain knowledge. The segmenter relied on the PMI information to the recover 85.7 the test poems. We could segment a poem completely correct only 17.8 time, however. When we presented our work at DH 2019, we have annotated more than 20000 poems. With a much larger amount of data, we were able to apply biLSTM models for this word segmentation task, and we segmented a poem completely correct above 20 completely agreed on their annotations about 40



There are no comments yet.


page 4


Towards Accurate Word Segmentation for Chinese Patents

A patent is a property right for an invention granted by the government ...

Word Segmentation as Graph Partition

We propose a new approach to the Chinese word segmentation problem that ...

Active Learning for Chinese Word Segmentation in Medical Text

Electronic health records (EHRs) stored in hospital information systems ...

SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining

Conventional tokenization methods for Chinese pretrained language models...

Hyperbolic Deep Learning for Chinese Natural Language Understanding

Recently hyperbolic geometry has proven to be effective in building embe...

CASICT Tibetan Word Segmentation System for MLWS2017

We participated in the MLWS 2017 on Tibetan word segmentation task, our ...

CASCONet: A Conference dataset

Knowledge mobilization and translation describes the process of moving k...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.