DeepAI
Log In Sign Up

Onto Word Segmentation of the Complete Tang Poems

08/28/2019
by   Chao-Lin Liu, et al.
0

We aim at segmenting words in the Complete Tang Poems (CTP). Although it is possible to do some research about CTP without doing full-scale word segmentation, we must move forward to word-level analysis of CTP for conducting advanced research topics. In November 2018 when we submitted the manuscript for DH 2019 (ADHO), we collected only 2433 poems that were segmented by trained experts, and used the segmented poems to evaluate the segmenter that considered domain knowledge of Chinese poetry. We trained pointwise mutual information (PMI) between Chinese characters based on the CTP poems (excluding the 2433 poems, which were used exclusively only for testing) and the domain knowledge. The segmenter relied on the PMI information to the recover 85.7 the test poems. We could segment a poem completely correct only 17.8 time, however. When we presented our work at DH 2019, we have annotated more than 20000 poems. With a much larger amount of data, we were able to apply biLSTM models for this word segmentation task, and we segmented a poem completely correct above 20 completely agreed on their annotations about 40

READ FULL TEXT
11/30/2016

Towards Accurate Word Segmentation for Chinese Patents

A patent is a property right for an invention granted by the government ...
04/05/2018

Word Segmentation as Graph Partition

We propose a new approach to the Chinese word segmentation problem that ...
08/22/2019

Active Learning for Chinese Word Segmentation in Medical Text

Electronic health records (EHRs) stored in hospital information systems ...
10/31/2022

Mining Word Boundaries in Speech as Naturally Annotated Word Segmentation Data

Chinese word segmentation (CWS) models have achieved very high performan...
10/25/2022

Topical Segmentation of Spoken Narratives: A Test Case on Holocaust Survivor Testimonies

The task of topical segmentation is well studied, but previous work has ...
10/17/2017

CASICT Tibetan Word Segmentation System for MLWS2017

We participated in the MLWS 2017 on Tibetan word segmentation task, our ...
05/21/2019

A realistic and robust model for Chinese word segmentation

A realistic Chinese word segmentation tool must adapt to textual variati...