Improving Document Clustering by Eliminating Unnatural Language

03/16/2017
by   Myungha Jang, et al.
University of Massachusetts Amherst
Emory University
0

Technical documents contain a fair amount of unnatural language, such as tables, formulas, pseudo-codes, etc. Unnatural language can be an important factor of confusing existing NLP tools. This paper presents an effective method of distinguishing unnatural language from natural language, and evaluates the impact of unnatural language detection on NLP tasks such as document clustering. We view this problem as an information extraction task and build a multiclass classification model identifying unnatural language components into four categories. First, we create a new annotated corpus by collecting slides and papers in various formats, PPT, PDF, and HTML, where unnatural language components are annotated into four categories. We then explore features available from plain text to build a statistical model that can handle any format as long as it is converted into plain text. Our experiments show that removing unnatural language components gives an absolute improvement in document clustering up to 15

READ FULL TEXT

page 1

page 2

page 3

page 4

11/01/2016

Recurrent Neural Network Language Model Adaptation Derived Document Vector

In many natural language processing (NLP) tasks, a document is commonly ...
05/25/2023

Neural Natural Language Processing for Long Texts: A Survey of the State-of-the-Art

The adoption of Deep Neural Networks (DNNs) has greatly benefited Natura...
10/06/2017

Czech Text Document Corpus v 2.0

This paper introduces "Czech Text Document Corpus v 2.0", a collection o...
01/02/2021

Cross-Document Language Modeling

We introduce a new pretraining approach for language models that are gea...
01/08/2022

Clustering Text Using Attention

Clustering Text has been an important problem in the domain of Natural L...
09/19/2020

Nominal Compound Chain Extraction: A New Task for Semantic-enriched Lexical Chain

Lexical chain consists of cohesion words in a document, which implies th...

Please sign up or login with your details

Forgot password? Click here to reset