Validation and Normalization of DCS corpus using Sanskrit Heritage tools to build a tagged Gold Corpus

05/13/2020
by   Sriram Krishnan, et al.
0

The Digital Corpus of Sanskrit records around 650,000 sentences along with their morphological and lexical tagging. But inconsistencies in morphological analysis, and in providing crucial information like the segmented word, urges the need for standardization and validation of this corpus. Automating the validation process requires efficient analyzers which also provide the missing information. The Sanskrit Heritage Engine's Reader produces all possible segmentations with morphological and lexical analyses. Aligning these systems would help us in recording the linguistic differences, which can be used to update these systems to produce standardized results and will also provide a Gold corpus tagged with complete morphological and lexical information along with the segmented words. Krishna et al. (2017) aligned 115,000 sentences, considering some of the linguistic differences. As both these systems have evolved significantly, the alignment is done again considering all the remaining linguistic differences between these systems. This paper describes the modified alignment process in detail and records the additional linguistic differences observed. Reference: Amrith Krishna, Pavankumar Satuluri, and Pawan Goyal. 2017. A dataset for Sanskrit word segmentation. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, page 105-114. Association for Computational Linguistics, August.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/08/2021

User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization

Morphological analysis (MA) and lexical normalization (LN) are both impo...
research
05/31/2021

How Lexical Gold Standards Have Effects On The Usefulness Of Text Analysis Tools For Digital Scholarship

This paper describes how the current lexical similarity and analogy gold...
research
10/06/2015

Analyzer and generator for Pali

This work describes a system that performs morphological analysis and ge...
research
03/09/2016

Lexical bundles in computational linguistics academic literature

In this study we analyzed a corpus of 8 million words academic literatur...
research
08/13/2018

Comparing morphological complexity of Spanish, Otomi and Nahuatl

We use two small parallel corpora for comparing the morphological comple...
research
03/25/2017

Morphological Analysis for the Maltese Language: The Challenges of a Hybrid System

Maltese is a morphologically rich language with a hybrid morphological s...
research
07/11/2022

TArC: Tunisian Arabish Corpus First complete release

In this paper we present the final result of a project on Tunisian Arabi...

Please sign up or login with your details

Forgot password? Click here to reset