Asking questions on handwritten document collections

by   Minesh Mathew, et al.

This work addresses the problem of Question Answering (QA) on handwritten document collections. Unlike typical QA and Visual Question Answering (VQA) formulations where the answer is a short text, we aim to locate a document snippet where the answer lies. The proposed approach works without recognizing the text in the documents. We argue that the recognition-free approach is suitable for handwritten documents and historical collections where robust text recognition is often difficult. At the same time, for human users, document image snippets containing answers act as a valid alternative to textual answers. The proposed approach uses an off-the-shelf deep embedding network which can project both textual words and word images into a common sub-space. This embedding bridges the textual and visual domains and helps us retrieve document snippets that potentially answer a question. We evaluate results of the proposed approach on two new datasets: (i) HW-SQuAD: a synthetic, handwritten document image counterpart of SQuAD1.0 dataset and (ii) BenthamQA: a smaller set of QA pairs defined on documents from the popular Bentham manuscripts collection. We also present a thorough analysis of the proposed recognition-free approach compared to a recognition-based approach which uses text recognized from the images using an OCR. Datasets presented in this work are available to download at


page 7

page 9


Recognition-free Question Answering on Handwritten Document Collections

In recent years, considerable progress has been made in the research are...

PageNet: Page Boundary Extraction in Historical Handwritten Documents

When digitizing a document into an image, it is common to include a surr...

A Probabilistic Framework for Lexicon-based Keyword Spotting in Handwritten Text Images

Query by String Keyword Spotting (KWS) is here considered as a key techn...

Open Set Classification of Untranscribed Handwritten Documents

Huge amounts of digital page images of important manuscripts are preserv...

BAGS: An automatic homework grading system using the pictures taken by smart phones

Homework grading is critical to evaluate teaching quality and effect. Ho...

Fused Text Recogniser and Deep Embeddings Improve Word Recognition and Retrieval

Recognition and retrieval of textual content from the large document col...

Measuring Human Perception to Improve Handwritten Document Transcription

The subtleties of human perception, as measured by vision scientists thr...

Please sign up or login with your details

Forgot password? Click here to reset