Predicting the Ordering of Characters in Japanese Historical Documents

by   Alex Lamb, et al.

Japan is a unique country with a distinct cultural heritage, which is reflected in billions of historical documents that have been preserved. However, the change in Japanese writing system in 1900 made these documents inaccessible for the general public. A major research project has been to make these historical documents accessible and understandable. An increasing amount of research has focused on the character recognition task and the location of characters on image, yet less research has focused on how to predict the sequential ordering of the characters. This is because sequence in classical Japanese is very different from modern Japanese. Ordering characters into a sequence is important for making the document text easily readable and searchable. Additionally, it is a necessary step for any kind of natural language processing on the data (e.g. machine translation, language modeling, and word embeddings). We explore a few approaches to the task of predicting the sequential ordering of the characters: one using simple hand-crafted rules, another using hand-crafted rules with adaptive thresholds, and another using a deep recurrent sequence model trained with teacher forcing. We provide a quantitative and qualitative comparison of these techniques as well as their distinct trade-offs. Our best-performing system has an accuracy of 98.65% and has a perfect accuracy on 49% of the books in our dataset, suggesting that the technique is able to predict the order of the characters well enough for many tasks.


page 4

page 12


KuroNet: Pre-Modern Japanese Kuzushiji Character Recognition with Deep Learning

Kuzushiji, a cursive writing style, had been used in Japan for over a th...

HRCenterNet: An Anchorless Approach to Chinese Character Segmentation in Historical Documents

The information provided by historical documents has always been indispe...

Modernizing Historical Documents: a User Study

Accessibility to historical documents is mostly limited to scholars. Thi...

Categorizing ancient documents

The analysis of historical documents is still a topical issue given the ...

Degraded Historical Documents Images Binarization Using a Combination of Enhanced Techniques

Document image binarization is the initial step and a crucial in many do...

Encryption by using base-n systems with many characters

It is possible to interpret text as numbers (and vice versa) if one inte...

uniblock: Scoring and Filtering Corpus with Unicode Block Information

The preprocessing pipelines in Natural Language Processing usually invol...

Please sign up or login with your details

Forgot password? Click here to reset