[论文解读] Probabilistic Modelling of Morphologically Rich Languages

Jan A. Botha|arXiv (Cornell University)|Jan 1, 2014

Natural Language Processing Techniques参考文献 177被引用 4

一句话总结

本论文提出了一种概率语言模型，通过显式引入词形丰富的语言中的词素子结构，利用贝叶斯n-gram模型和分布式词素表征，以改善平滑化和泛化能力。该研究提出了一种新颖的无监督模型，用于发现连续和非连续词素，并在词形分割和机器翻译等下游任务中展现出性能提升。

ABSTRACT

This thesis investigates how the sub-structure of words can be accounted for in probabilistic models of language. Such models play an important role in natural language processing tasks such as translation or speech recognition, but often rely on the simplistic assumption that words are opaque symbols. This assumption does not fit morphologically complex language well, where words can have rich internal structure and sub-word elements are shared across distinct word forms. Our approach is to encode basic notions of morphology into the assumptions of three different types of language models, with the intention that leveraging shared sub-word structure can improve model performance and help overcome data sparsity that arises from morphological processes. In the context of n-gram language modelling, we formulate a new Bayesian model that relies on the decomposition of compound words to attain better smoothing, and we develop a new distributed language model that learns vector representations of morphemes and leverages them to link together morphologically related words. In both cases, we show that accounting for word sub-structure improves the models' intrinsic performance and provides benefits when applied to other tasks, including machine translation. We then shift the focus beyond the modelling of word sequences and consider models that automatically learn <em>what</em> the sub-word elements of a given language are, given an unannotated list of words. We formulate a novel model that can learn discontiguous morphemes in addition to the more conventional contiguous morphemes that most previous models are limited to. This approach is demonstrated on Semitic languages, and we find that modelling discontiguous sub-word structures leads to improvements in the task of segmenting words into their contiguous morphemes.

研究动机与目标

• 通过利用子词结构，解决词形丰富语言语言模型中的数据稀疏性问题。
• 通过复合词的贝叶斯分解，改善n-gram模型的平滑化和泛化能力。
• 开发学习词素向量表征的分布式语言模型，以关联相关词形。
• 从未标注的词表中无监督发现词形单位，包括非连续词素。
• 在词形分割和机器翻译等任务中，展示子词建模的实用性。

提出的方法

• 提出一种贝叶斯n-gram模型，将复合词分解为词素，以增强平滑化并处理数据稀疏性。
• 设计一种分布式语言模型，学习词素的密集向量表征，并利用其建模词形之间的相似性。
• 提出一种新颖的概率模型，用于无监督词素发现，显式处理非连续词素。
• 将该模型应用于闪米特语系，其词形复杂且非连续词素常见。
• 使用变分推断和非参数贝叶斯方法估计模型参数并推断词形结构。
• 将学习到的词素表征集成到下游自然语言处理任务中，如机器翻译和词形分割。

实验结果

研究问题

• RQ1 建模词的子结构是否能提升内在语言模型性能和数据效率？
• RQ2 分布式词素表征在多大程度上能增强对词形相关词的泛化能力？
• RQ3 概率模型能否从未标注的词表中发现词形复杂语言中的非连续词素？
• RQ4 子结构建模如何影响词形分割和机器翻译任务的性能？
• RQ5 在n-gram模型中引入词形分解是否能带来更好的平滑化和更低的困惑度？

主要发现

• 与标准n-gram模型相比，采用词形分解的贝叶斯n-gram模型实现了更好的平滑化和更低的困惑度。
• 使用学习到的词素向量的分布式语言模型在词形丰富的测试集上表现出更强的泛化能力，并取得更低的困惑度。
• 所提出的无监督模型成功识别了闪米特语系中的连续和非连续词素，优于仅限于连续单位的模型。
• 当将发现的词素结构作为先验或特征输入时，词形分割性能得到提升。
• 将具有词形意识的语言模型集成到神经机器翻译系统中，带来了可测量的性能改进。
• 该模型在多种词形范式中表现出鲁棒性，尤其在黏着语和屈折语中表现突出。