Extracting and Validating Explanatory Word Archipelagoes using Dual Entropy

02/22/2020
by   Yukio Ohsawa, et al.
0

The logical connectivity of text is represented by the connectivity of words that form archipelagoes. Here, each archipelago is a sequence of islands of the occurrences of a certain word. An island here means the local sequence of sentences where the word is emphasized, and an archipelago of a length comparable to the target text is extracted using the co-variation of entropy A (the window-based entropy) on the distribution of the word's occurrences with the width of each time window. Then, the logical connectivity of text is evaluated on entropy B (the graph-based entropy) computed on the distribution of sentences to connected word-clusters obtained on the co-occurrence of words. The results show the parts of the target text with words forming archipelagoes extracted on entropy A, without learned or prepared knowledge, form an explanatory part of the text that is of smaller entropy B than the parts extracted by the baseline methods.

READ FULL TEXT

Authors

page 1

page 2

page 3

page 4

10/17/2017

CASICT Tibetan Word Segmentation System for MLWS2017

We participated in the MLWS 2017 on Tibetan word segmentation task, our ...
01/06/2014

Bangla Text Recognition from Video Sequence: A New Focus

Extraction and recognition of Bangla text from video frame images is cha...
02/07/2022

The Input and Output Entropies of the k-Deletion/Insertion Channel

The channel output entropy of a transmitted word is the entropy of the p...
06/20/2022

SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluation

Word Mover's Distance (WMD) computes the distance between words and mode...
04/18/2020

Effect of Text Color on Word Embeddings

In natural scenes and documents, we can find the correlation between a t...
05/23/2022

Information Propagation by Composited Labels in Natural Language Processing

In natural language processing (NLP), labeling on regions of text, such ...
10/12/2015

Asymptotic Logical Uncertainty and The Benford Test

We give an algorithm A which assigns probabilities to logical sentences....
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.