uniblock: Scoring and Filtering Corpus with Unicode Block Information

08/26/2019
by   Yingbo Gao, et al.
0

The preprocessing pipelines in Natural Language Processing usually involve a step of removing sentences consisted of illegal characters. The definition of illegal characters and the specific removal strategy depend on the task, language, domain, etc, which often lead to tiresome and repetitive scripting of rules. In this paper, we introduce a simple statistical method, uniblock, to overcome this problem. For each sentence, uniblock generates a fixed-size feature vector using Unicode block information of the characters. A Gaussian mixture model is then estimated on some clean corpus using variational inference. The learned model can then be used to score sentences and filter corpus. We present experimental results on Sentiment Analysis, Language Modeling and Machine Translation, and show the simplicity and effectiveness of our method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/13/2018

Demo of Sanskrit-Hindi SMT System

The demo proposal presents a Phrase-based Sanskrit-Hindi (SaHiT) Statist...
research
11/10/2015

Information retrieval in folktales using natural language processing

Our aim is to extract information about literary characters in unstructu...
research
07/28/2020

Preparation of Sentiment tagged Parallel Corpus and Testing its effect on Machine Translation

In the current work, we explore the enrichment in the machine translatio...
research
05/22/2011

Correction of Noisy Sentences using a Monolingual Corpus

Correction of Noisy Natural Language Text is an important and well studi...
research
05/22/2020

Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation

Black-box machine translation systems have proven incredibly useful for ...
research
06/12/2021

Predicting the Ordering of Characters in Japanese Historical Documents

Japan is a unique country with a distinct cultural heritage, which is re...
research
03/11/2021

Learning Feature Weights using Reward Modeling for Denoising Parallel Corpora

Large web-crawled corpora represent an excellent resource for improving ...

Please sign up or login with your details

Forgot password? Click here to reset