Extracting and Validating Explanatory Word Archipelagoes using Dual Entropy

by   Yukio Ohsawa, et al.

The logical connectivity of text is represented by the connectivity of words that form archipelagoes. Here, each archipelago is a sequence of islands of the occurrences of a certain word. An island here means the local sequence of sentences where the word is emphasized, and an archipelago of a length comparable to the target text is extracted using the co-variation of entropy A (the window-based entropy) on the distribution of the word's occurrences with the width of each time window. Then, the logical connectivity of text is evaluated on entropy B (the graph-based entropy) computed on the distribution of sentences to connected word-clusters obtained on the co-occurrence of words. The results show the parts of the target text with words forming archipelagoes extracted on entropy A, without learned or prepared knowledge, form an explanatory part of the text that is of smaller entropy B than the parts extracted by the baseline methods.



page 1

page 2

page 3

page 4


CASICT Tibetan Word Segmentation System for MLWS2017

We participated in the MLWS 2017 on Tibetan word segmentation task, our ...

Bangla Text Recognition from Video Sequence: A New Focus

Extraction and recognition of Bangla text from video frame images is cha...

The Input and Output Entropies of the k-Deletion/Insertion Channel

The channel output entropy of a transmitted word is the entropy of the p...

SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluation

Word Mover's Distance (WMD) computes the distance between words and mode...

Effect of Text Color on Word Embeddings

In natural scenes and documents, we can find the correlation between a t...

Information Propagation by Composited Labels in Natural Language Processing

In natural language processing (NLP), labeling on regions of text, such ...

Asymptotic Logical Uncertainty and The Benford Test

We give an algorithm A which assigns probabilities to logical sentences....
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.