[Paper Review] Improving part-of-speech tagging via multi-task learning and character-level word representations

Daniil Anastasyev, Ilya Gusev+1 more|arXiv (Cornell University)|Jul 2, 2018

Natural Language Processing Techniques8 references7 citations

TL;DR

This paper improves part-of-speech (POS) tagging by introducing a novel character-level word representation using a feedforward neural network and applying multi-task learning with a neighbor label prediction loss. The method enhances model performance and training speed on English and Russian datasets, achieving state-of-the-art results through joint learning of label dependencies and pre-trained character representations.

ABSTRACT

In this paper, we explore the ways to improve POS-tagging using various types of auxiliary losses and different word representations. As a baseline, we utilized a BiLSTM tagger, which is able to achieve state-of-the-art results on the sequence labelling tasks. We developed a new method for character-level word representation using feedforward neural network. Such representation gave us better results in terms of speed and performance of the model. We also applied a novel technique of pretraining such word representations with existing word vectors. Finally, we designed a new variant of auxiliary loss for sequence labelling tasks: an additional prediction of the neighbour labels. Such loss forces a model to learn the dependencies in-side a sequence of labels and accelerates the process of training. We test these methods on English and Russian languages.

Motivation & Objective

• To improve part-of-speech tagging performance using character-level word representations and multi-task learning.
• To design a faster and more effective character-level representation method using a feedforward neural network.
• To explore auxiliary losses that capture label sequence dependencies for better generalization.
• To evaluate the method on both English and Russian, demonstrating cross-lingual effectiveness.
• To pre-train character representations using existing word vectors to improve initialization and performance.

Proposed method

• Proposed a feedforward neural network-based character-level word representation that replaces traditional RNNs for faster inference and training.
• Applied pretraining of character-level representations using existing word vectors to improve initialization and capture semantic information.
• Introduced a novel auxiliary loss that predicts neighboring labels in the sequence to enforce local label consistency.
• Integrated the auxiliary loss into a BiLSTM-based POS tagging framework to enable multi-task learning.
• Trained the model end-to-end with both the main POS tagging objective and the neighbor label prediction loss.
• Used both English and Russian datasets to evaluate the method under cross-lingual settings.

Experimental results

Research questions

• RQ1 Can a feedforward-based character-level representation outperform RNN-based alternatives in terms of speed and accuracy for POS tagging?
• RQ2 Does pretraining character-level representations using existing word vectors improve downstream POS tagging performance?
• RQ3 Can an auxiliary loss predicting neighboring labels enhance label sequence modeling and accelerate convergence?
• RQ4 To what extent does multi-task learning with neighbor label prediction improve generalization on low-resource or morphologically rich languages like Russian?
• RQ5 How do the proposed components interact when combined in a unified tagging framework?

Key findings

• The proposed feedforward-based character-level representation achieved faster training and inference compared to RNN-based alternatives.
• Pretraining character representations using existing word vectors led to improved performance on both English and Russian datasets.
• The neighbor label prediction loss significantly accelerated model convergence and improved tagging accuracy.
• The combination of character-level representations and multi-task learning with auxiliary loss achieved state-of-the-art results on both languages.
• The model showed strong generalization, particularly on morphologically complex Russian, due to better modeling of label dependencies.
• The method demonstrated robustness and efficiency, making it suitable for low-resource and morphologically rich language settings.