Subjective Assessments of Legibility in Ancient Manuscript Images – The SALAMI Dataset

02/19/2021
by   Simon Brenner, et al.
0

The research field concerned with the digital restoration of degraded written heritage lacks a quantitative metric for evaluating its results, which prevents the comparison of relevant methods on large datasets. Thus, we introduce a novel dataset of Subjective Assessments of Legibility in Ancient Manuscript Images (SALAMI) to serve as a ground truth for the development of quantitative evaluation metrics in the field of digital text restoration. This dataset consists of 250 images of 50 manuscript regions with corresponding spatial maps of mean legibility and uncertainty, which are based on a study conducted with 20 experts of philology and paleography. As this study is the first of its kind, the validity and reliability of its design and the results obtained are motivated statistically: we report a high intra- and inter-rater agreement and show that the bulk of variation in the scores is introduced by the images regions observed and not by controlled or uncontrolled properties of participants and test environments, thus concluding that the legibility scores measured are valid attributes of the underlying images.

READ FULL TEXT

page 3

page 4

page 5

page 7

research
09/22/2017

Novel Evaluation Metrics for Seam Carving based Image Retargeting

Image retargeting effectively resizes images by preserving the recogniza...
research
08/01/2017

A Locally Weighted Fixation Density-Based Metric for Assessing the Quality of Visual Saliency Predictions

With the increased focus on visual attention (VA) in the last decade, a ...
research
12/31/2022

Approaching Peak Ground Truth

Machine learning models are typically evaluated by computing similarity ...
research
09/02/2018

Modeling Topical Coherence in Discourse without Supervision

Coherence of text is an important attribute to be measured for both manu...
research
02/13/2022

An Analysis of Variations in the Effectiveness of Query Performance Prediction

A query performance predictor estimates the retrieval effectiveness of a...
research
04/07/2022

MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aids

Improving the user's hearing ability to understand speech in noisy envir...
research
09/28/2020

Describing Subjective Experiment Consistency by p-Value P-P Plot

There are phenomena that cannot be measured without subjective testing. ...

Please sign up or login with your details

Forgot password? Click here to reset