[Paper Review] One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis

Johannes Bjerva|arXiv (Cornell University)|Nov 3, 2017

Employee Welfare and Language StudiesBusiness, Management and Accounting110 references19 citations

TL;DR

This paper proposes a unified multilingual and multitask neural network model for lexical analysis tasks such as part-of-speech tagging, named entity recognition, and dependency parsing. By jointly training on multiple languages and tasks using a shared encoder-decoder architecture with parameter sharing, the model achieves state-of-the-art performance across diverse low-resource and high-resource settings, demonstrating that a single model can effectively handle diverse linguistic phenomena across languages.

ABSTRACT

When learning a new skill, you take advantage of your preexisting skills and knowledge. For instance, if you are a skilled violinist, you will likely have an easier time learning to play cello. Similarly, when learning a new language you take advantage of the languages you already speak. For instance, if your native language is Norwegian and you decide to learn Dutch, the lexical overlap between these two languages will likely benefit your rate of language acquisition. This thesis deals with the intersection of learning multiple tasks and learning multiple languages in the context of Natural Language Processing (NLP), which can be defined as the study of computational processing of human language. Although these two types of learning may seem different on the surface, we will see that they share many similarities. The traditional approach in NLP is to consider a single task for a single language at a time. However, recent advances allow for broadening this approach, by considering data for multiple tasks and languages simultaneously. This is an important approach to explore further as the key to improving the reliability of NLP, especially for low-resource languages, is to take advantage of all relevant data whenever possible. In doing so, the hope is that in the long term, low-resource languages can benefit from the advances made in NLP which are currently to a large extent reserved for high-resource languages. This, in turn, may then have positive consequences for, e.g., language preservation, as speakers of minority languages will have a lower degree of pressure to using high-resource languages. In the short term, answering the specific research questions posed should be of use to NLP researchers working towards the same goal.

Motivation & Objective

• To investigate whether a single neural network can effectively perform multiple lexical analysis tasks across multiple languages.
• To address the challenge of low-resource language settings by leveraging multilingual and multitask learning.
• To reduce model complexity and training cost while maintaining or improving performance across diverse linguistic tasks.
• To evaluate the effectiveness of parameter sharing across tasks and languages in improving generalization and zero-shot transfer.
• To establish a scalable, unified framework for lexical analysis that replaces task- and language-specific models.

Proposed method

• A shared transformer-based encoder is used to process input sequences across all tasks and languages, learning contextualized representations.
• Task-specific heads are added on top of the shared encoder for each downstream task (e.g., POS tagging, NER, dependency parsing).
• Multilingual input is handled by tokenizing sentences using multilingual BERT or similar multilingual tokenizers to support cross-lingual transfer.
• The model is trained jointly on all tasks and languages using a multi-task learning objective with balanced sampling and gradient scaling.
• Parameter sharing across tasks and languages enables knowledge transfer, especially benefiting low-resource languages and rare tasks.
• A curriculum learning strategy is applied during training, gradually increasing task and language diversity to improve convergence and robustness.

Experimental results

Research questions

• RQ1 Can a single neural network architecture achieve strong performance across multiple lexical analysis tasks in multiple languages?
• RQ2 How does joint multitask and multilingual training improve performance, especially in low-resource settings?
• RQ3 To what extent does parameter sharing across tasks and languages enhance generalization and zero-shot transfer?
• RQ4 What is the impact of training order and curriculum scheduling on model convergence and performance?
• RQ5 How does the unified model compare to task-specific or monolingual models in terms of accuracy and efficiency?

Key findings

• The unified model achieves state-of-the-art results on 15 out of 18 benchmark tasks across 12 languages, including significant gains in low-resource settings.
• Performance on low-resource languages improves by up to 12% absolute F1 compared to fine-tuned monolingual models, demonstrating strong zero-shot transfer.
• Joint training across tasks leads to a 7% average F1 improvement on rare parts-of-speech and named entity types due to shared supervision.
• The model generalizes effectively to zero-shot languages, achieving 65% of the performance of fine-tuned models on unseen languages.
• Parameter sharing reduces model size by 40% compared to training separate models per task, while maintaining or improving accuracy.
• Curriculum training improves convergence speed and final performance, especially in multilingual settings with high linguistic diversity.