Segmenting Natural Language Sentences via Lexical Unit Analysis

12/10/2020
by   Yangming Li, et al.
0

In this work, we present Lexical Unit Analysis (LUA), a framework for general sequence segmentation tasks. Given a natural language sentence, LUA scores all the valid segmentation candidates and utilizes dynamic programming (DP) to extract the maximum scoring one. LUA enjoys a number of appealing properties such as inherently guaranteeing the predicted segmentation to be valid and facilitating globally optimal training and inference. Besides, the practical time complexity of LUA can be reduced to linear time, which is very efficient. We have conducted extensive experiments on 5 tasks, including syntactic chunking, named entity recognition (NER), slot filling, Chinese word segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our models have achieved the state-of-the-art performances on 13 of them. The results also show that the F1 score of identifying long-length segments is notably improved.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/11/2021

A More Efficient Chinese Named Entity Recognition base on BERT and Syntactic Analysis

We propose a new Named entity recognition (NER) method to effectively ma...
research
04/15/2021

Neural Sequence Segmentation as Determining the Leftmost Segments

Prior methods to text segmentation are mostly at token level. Despite th...
research
07/05/2018

Chinese Lexical Analysis with Deep Bi-GRU-CRF Network

Lexical analysis is believed to be a crucial step towards natural langua...
research
12/16/2021

Simple Questions Generate Named Entity Recognition Datasets

Named entity recognition (NER) is a task of extracting named entities of...
research
04/14/2022

Qtrade AI at SemEval-2022 Task 11: An Unified Framework for Multilingual NER Task

This paper describes our system, which placed third in the Multilingual ...
research
05/31/2023

A Global Context Mechanism for Sequence Labeling

Sequential labeling tasks necessitate the computation of sentence repres...
research
04/08/2015

Exploring Lexical, Syntactic, and Semantic Features for Chinese Textual Entailment in NTCIR RITE Evaluation Tasks

We computed linguistic information at the lexical, syntactic, and semant...

Please sign up or login with your details

Forgot password? Click here to reset