DeepAI
Log In Sign Up

Predicting the Ordering of Characters in Japanese Historical Documents

06/12/2021
by   Alex Lamb, et al.
0

Japan is a unique country with a distinct cultural heritage, which is reflected in billions of historical documents that have been preserved. However, the change in Japanese writing system in 1900 made these documents inaccessible for the general public. A major research project has been to make these historical documents accessible and understandable. An increasing amount of research has focused on the character recognition task and the location of characters on image, yet less research has focused on how to predict the sequential ordering of the characters. This is because sequence in classical Japanese is very different from modern Japanese. Ordering characters into a sequence is important for making the document text easily readable and searchable. Additionally, it is a necessary step for any kind of natural language processing on the data (e.g. machine translation, language modeling, and word embeddings). We explore a few approaches to the task of predicting the sequential ordering of the characters: one using simple hand-crafted rules, another using hand-crafted rules with adaptive thresholds, and another using a deep recurrent sequence model trained with teacher forcing. We provide a quantitative and qualitative comparison of these techniques as well as their distinct trade-offs. Our best-performing system has an accuracy of 98.65% and has a perfect accuracy on 49% of the books in our dataset, suggesting that the technique is able to predict the order of the characters well enough for many tasks.

READ FULL TEXT

page 4

page 12

10/21/2019

KuroNet: Pre-Modern Japanese Kuzushiji Character Recognition with Deep Learning

Kuzushiji, a cursive writing style, had been used in Japan for over a th...
12/10/2020

HRCenterNet: An Anchorless Approach to Chinese Character Segmentation in Historical Documents

The information provided by historical documents has always been indispe...
07/01/2019

Modernizing Historical Documents: a User Study

Accessibility to historical documents is mostly limited to scholars. Thi...
08/28/2013

Categorizing ancient documents

The analysis of historical documents is still a topical issue given the ...
01/27/2019

Degraded Historical Documents Images Binarization Using a Combination of Enhanced Techniques

Document image binarization is the initial step and a crucial in many do...
08/26/2019

uniblock: Scoring and Filtering Corpus with Unicode Block Information

The preprocessing pipelines in Natural Language Processing usually invol...