[论文解读] Basic Linguistic Resources and Baselines for Bhojpuri, Magahi and Maithili for Natural Language Processing

Rajesh Kumar Mundotiya, Manish Kumar Singh+4 more|arXiv (Cornell University)|Jan 1, 2020

Natural Language Processing Techniques参考文献 64被引用 4

一句话总结

本论文为印度普尔瓦恩查勒地区低资源的印度-雅利安语——博杰布尔语、马拉吉语和迈蒂利语——编制了全新整理、清洗并进行语言学标注的语料库，使用BIS词性标注集对词性（POS）和短语块进行标注。论文在字符、单词、音节和词素层面提供了对比性语言学统计数据，为自然语言处理（NLP）开发提供了基础资源与基线数据，尽管原始语料规模存在固有差异，但通过调整语料规模以实现公平比较。

ABSTRACT

Corpus preparation for low-resource languages and for development of human language technology to analyze or computationally process them is a laborious task, primarily due to the unavailability of expert linguists who are native speakers of these languages and also due to the time and resources required. Bhojpuri, Magahi, and Maithili, languages of the Purvanchal region of India (in the north-eastern parts), are low-resource languages belonging to the Indo-Aryan (or Indic) family. They are closely related to Hindi, which is a relatively high-resource language, which is why we make our comparisons with Hindi. We collected corpora for these three languages from various sources and cleaned them to the extent possible, without changing the data in them. The text belongs to different domains and genres. We calculated some basic statistical measures for these corpora at character, word, syllable, and morpheme levels. These corpora were also annotated with parts-of-speech (POS) and chunk tags. The basic statistical measures were both absolute and relative and were meant to give an indication of linguistic properties such as morphological, lexical, phonological, and syntactic complexities (or richness). The results were compared with a standard Hindi corpus. For most of the measures, we tried to keep the size of the corpus the same across the languages so as to avoid the effect of corpus size, but in some cases it turned out that using the full corpus was better, even if sizes were very different. Although the results are not very clear, we try to draw some conclusions about the languages and the corpora. For POS tagging and chunking, the BIS tagset was used to manually annotate the data. The sizes of the POS tagged data are 16067, 14669 and 12310 sentences, respectively for Bhojpuri, Magahi and Maithili. The sizes for chunking are 9695 and 1954 sentences for Bhojpuri and Maithili, respect

研究动机与目标

• 为解决印度普尔瓦恩查勒地区低资源印度-雅利安语缺乏语言学资源的问题。
• 从多个来源收集并清洗涵盖多样化领域的博杰布尔语、马拉吉语和迈蒂利语语料库，同时不改变原始内容。
• 在字符、单词、音节和词素层面计算基本语言学统计数据，以评估形态学、词汇、音系学和句法复杂性。
• 使用BIS标注集手动标注词性（POS）和短语块标签，为下游NLP任务提供一致的语言学评估。
• 在可能的情况下，通过最小化语料规模偏差，比较这些语言与标准印地语语料库之间的语言属性和资源特征。

提出的方法

• 从多种来源收集博杰布尔语、马拉吉语和迈蒂利语的原始文本语料库，确保领域和体裁的多样性。
• 执行数据清洗，在不改变语言内容的前提下提升可用性。
• 在字符、单词、音节和词素层面计算绝对和相对的语言学统计数据，以评估语言复杂性。
• 使用BIS标注集对语料库进行词性（POS）和短语块标签的标注，以实现一致的语言学评估。
• 在可行范围内对各语言的语料库规模进行标准化，以便比较；但当更具有代表性时，保留完整的语料库规模。
• 与标准印地语语料库进行比较，以定位研究发现。

实验结果

研究问题

• RQ1 博杰布尔语、马拉吉语和迈蒂利语在形态学、词汇、音系学和句法复杂性方面与印地语相比如何？
• RQ2 博杰布尔语、马拉吉语和迈蒂利语语料库中的关键语言学统计数据（如词长、词素数量、音节结构）是什么？
• RQ3 语料库规模的差异在多大程度上影响了这些低资源语言之间语言比较的可靠性？
• RQ4 手动标注的词性（POS）和短语块标注数据集在这些语言未来NLP任务中的基线有效性如何？
• RQ5 从这些密切关联但资源匮乏的语言的语言学特性中，可以得出哪些见解？

主要发现

• 词性标注语料库包含博杰布尔语16,067个句子、马拉吉语14,669个句子和迈蒂利语12,310个句子，为NLP任务提供了大量训练数据。
• 短语块标注语料库包含博杰布尔语9,695个句子和迈蒂利语1,954个句子，支持句法分析与解析研究。
• 尽管努力平衡语料库规模，但原始语料库规模的差异仍影响了某些语言学指标的可比性。
• 在字符、单词、音节和词素等多个层面的语言学统计数据揭示了这三种语言在形态学和词汇复杂性方面的显著差异。
• 研究表明，即使资源有限，只要使用BIS标注集进行一致标注，也能为词性标注和短语块标注创建可靠的基线。
• 与印地语的对比分析显示，在语言丰富度和复杂性方面存在可测量的差异，表明每种语言都需要不同的NLP建模方法。