[Paper Review] Validation and Normalization of DCS corpus using Sanskrit Heritage tools to build a tagged Gold Corpus

Sriram Krishnan, Amba Kulkarni+1 more|arXiv (Cornell University)|May 13, 2020

Natural Language Processing Techniques11 references4 citations

TL;DR

This paper presents a refined alignment process between the Digital Corpus of Sanskrit (DCS) and the Sanskrit Heritage Engine to create a standardized, morphologically tagged, and segmented Gold corpus. By resolving linguistic discrepancies in lemma analysis, segmentation, and part-of-speech tagging—especially for compounds, derivatives, and indeclinables—it achieves 77% lemma mapping with 42% single mappings, significantly improving corpus quality for NLP tasks in Sanskrit.

ABSTRACT

The Digital Corpus of Sanskrit records around 650,000 sentences along with their morphological and lexical tagging. But inconsistencies in morphological analysis, and in providing crucial information like the segmented word, urges the need for standardization and validation of this corpus. Automating the validation process requires efficient analyzers which also provide the missing information. The Sanskrit Heritage Engine's Reader produces all possible segmentations with morphological and lexical analyses. Aligning these systems would help us in recording the linguistic differences, which can be used to update these systems to produce standardized results and will also provide a Gold corpus tagged with complete morphological and lexical information along with the segmented words. Krishna et al. (2017) aligned 115,000 sentences, considering some of the linguistic differences. As both these systems have evolved significantly, the alignment is done again considering all the remaining linguistic differences between these systems. This paper describes the modified alignment process in detail and records the additional linguistic differences observed. Reference: Amrith Krishna, Pavankumar Satuluri, and Pawan Goyal. 2017. A dataset for Sanskrit word segmentation. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, page 105-114. Association for Computational Linguistics, August.

Motivation & Objective

• To address inconsistencies in morphological analysis and word segmentation within the Digital Corpus of Sanskrit (DCS), which hinder reliable NLP applications.
• To standardize the DCS corpus by aligning it with the more linguistically detailed outputs of the Sanskrit Heritage Engine, ensuring consistent lemma, part-of-speech, and segmentation labeling.
• To identify and resolve linguistic discrepancies between the DCS annotation scheme and the Heritage Engine’s analysis, particularly for compounds, derivatives, and indeclinables.
• To produce a high-quality, tagged Gold corpus with complete morphological and lexical information, suitable for training and evaluating computational tools in Sanskrit.
• To reduce ambiguity and multiple mappings in lemma alignment by resolving differences in analysis between the two systems.

Proposed method

• Applying the Sanskrit Heritage Engine’s Reader to generate all possible segmentations and morphological analyses using finite-state technology and Effective Eilenberg machines.
• Performing a systematic alignment between DCS annotations and Heritage Engine outputs, focusing on lemma, part-of-speech, and morphological tags.
• Classifying and normalizing linguistic differences such as compound analysis, secondary derivatives (taddhitāntas), and indeclinables (e.g., api) to ensure consistent tagging.
• Prioritizing alignments based on linguistic plausibility and statistical frequency, while flagging unmatched or ambiguously mapped lemmas for manual review.
• Recording and categorizing discrepancies in analysis—such as participial vs. nominal forms (e.g., īśāna) or compound vs. derivative treatment (e.g., brahmamayī)—to inform corpus and tool improvements.
• Using a shared, compact representation of billions of possible segmentations to efficiently compare and map solutions, with statistical filtering to prioritize likely analyses.

Experimental results

Research questions

• RQ1 How can inconsistencies in morphological and lexical tagging within the DCS corpus be systematically identified and resolved?
• RQ2 What are the key linguistic discrepancies between the DCS annotation scheme and the Sanskrit Heritage Engine’s analysis, particularly for compounds and derivatives?
• RQ3 To what extent can the alignment process standardize lemma, part-of-speech, and segmentation labels across both systems?
• RQ4 How do differences in handling indeclinables (e.g., api) and morphological paradigms (e.g., participles vs. agent nouns) affect alignment accuracy?
• RQ5 What improvements in corpus quality and tool interoperability can be achieved through this alignment process?

Key findings

• Out of 119,502 sentences analyzed, 92,781 achieved full lemma mapping, representing a 77% success rate in alignment between DCS and the Sanskrit Heritage Engine.
• Of the successfully mapped sentences, 39,793 had exactly one matching lemma per word, indicating a 42% rate of unambiguous alignment.
• 12,639 sentences remained unmapped in full, with at least one lemma not found in the Heritage Engine, primarily due to missing lexical entries for words like prameyatvam and nirguṇatvam.
• Multiple mappings occurred in 52,988 sentences, mainly due to linguistic ambiguities such as compound vs. non-compound analysis (e.g., dvija-uttamaḥ vs. dvi-ja-uttamaḥ) or differing analyses of the same word (e.g., īśāna as participial or nominal form).
• The study identified 14,082 sentences that were modified but still had at least one unmapped lemma, highlighting persistent challenges in aligning secondary derivatives and morphological variants.
• Key issues included inconsistent treatment of taddhitāntas (e.g., brahmamayī as compound vs. derived word), morphological tagging differences (e.g., ‘iic’ vs. ‘pfp. iic.’), and misclassification of indeclinables (e.g., api as ‘ind.’ in DCS vs. ‘conj.’ or ‘prep.’ in Heritage).