[论文解读] One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis

Johannes Bjerva|arXiv (Cornell University)|Nov 3, 2017

Employee Welfare and Language StudiesBusiness, Management and Accounting参考文献 110被引用 19

一句话总结

本文提出了一种统一的多语言、多任务神经网络模型，用于词法分析任务，如词性标注、命名实体识别和依存句法分析。通过使用共享编码器-解码器架构并共享参数，联合在多种语言和任务上进行训练，该模型在多样化的低资源和高资源设置下均实现了最先进性能，表明单一模型能够有效处理跨语言的多种语言现象。

ABSTRACT

When learning a new skill, you take advantage of your preexisting skills and knowledge. For instance, if you are a skilled violinist, you will likely have an easier time learning to play cello. Similarly, when learning a new language you take advantage of the languages you already speak. For instance, if your native language is Norwegian and you decide to learn Dutch, the lexical overlap between these two languages will likely benefit your rate of language acquisition. This thesis deals with the intersection of learning multiple tasks and learning multiple languages in the context of Natural Language Processing (NLP), which can be defined as the study of computational processing of human language. Although these two types of learning may seem different on the surface, we will see that they share many similarities. The traditional approach in NLP is to consider a single task for a single language at a time. However, recent advances allow for broadening this approach, by considering data for multiple tasks and languages simultaneously. This is an important approach to explore further as the key to improving the reliability of NLP, especially for low-resource languages, is to take advantage of all relevant data whenever possible. In doing so, the hope is that in the long term, low-resource languages can benefit from the advances made in NLP which are currently to a large extent reserved for high-resource languages. This, in turn, may then have positive consequences for, e.g., language preservation, as speakers of minority languages will have a lower degree of pressure to using high-resource languages. In the short term, answering the specific research questions posed should be of use to NLP researchers working towards the same goal.

研究动机与目标

• 探究单一神经网络是否能有效在多种语言中执行多种词法分析任务。
• 通过利用多语言和多任务学习，解决低资源语言设置下的挑战。
• 在保持或提升多种语言任务性能的同时，降低模型复杂度和训练成本。
• 评估在任务和语言之间共享参数对泛化能力和零样本迁移的提升效果。
• 建立一个可扩展的统一框架，用于词法分析，替代针对特定任务或语言的模型。

提出的方法

• 使用共享的基于Transformer的编码器处理所有任务和语言的输入序列，学习上下文相关的表征。
• 在共享编码器之上为每个下游任务（如词性标注、命名实体识别、依存句法分析）添加特定任务的头。
• 通过使用多语言BERT或类似多语言分词器对多语言输入进行分词，以支持跨语言迁移。
• 使用平衡采样和梯度缩放的多任务学习目标，联合在所有任务和语言上进行训练。
• 在任务和语言之间共享参数，实现知识迁移，尤其有利于低资源语言和罕见任务。
• 在训练中应用课程学习策略，逐步增加任务和语言多样性，以提高收敛性和鲁棒性。

实验结果

研究问题

• RQ1 单一神经网络架构是否能在多种语言的多种词法分析任务中实现强大性能？
• RQ2 联合进行多任务和多语言训练在性能提升方面有何影响，特别是在低资源设置下？
• RQ3 在任务和语言之间共享参数在多大程度上增强了泛化能力和零样本迁移？
• RQ4 训练顺序和课程调度对模型收敛性和性能有何影响？
• RQ5 与特定任务或单语模型相比，统一模型在准确率和效率方面表现如何？

主要发现

• 该统一模型在18项基准任务中的15项上实现了最先进结果，涵盖12种语言，包括在低资源设置下的显著提升。
• 与微调后的单语模型相比，低资源语言的F1值最高提升12个百分点，表明具有强大的零样本迁移能力。
• 跨任务联合训练使罕见词性标注和命名实体类型的平均F1值提升7%，得益于共享监督。
• 该模型能有效泛化到零样本语言，在未见过的语言上达到微调模型65%的性能。
• 参数共享使模型大小相比每项任务单独训练的模型减少了40%，同时保持或提升了准确率。
• 课程训练提高了收敛速度和最终性能，尤其在语言多样性高的多语言设置中表现更优。