[Paper Review] Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks

Gor Arakelyan, Karen Hambardzumyan+1 more|arXiv (Cornell University)|Sep 10, 2018

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a joint neural network architecture using LSTMs to simultaneously predict lemmas, part-of-speech tags, and 21 morphological features for words in Universal Dependencies treebanks. The model extends a sequence tagging framework with character-level decoders and positional context, achieving competitive performance on nine languages despite not being fully optimized, marking a foundational step toward end-to-end joint dependency parsing and lemmatization.

ABSTRACT

This paper describes our submission to CoNLL 2018 UD Shared Task. We have extended an LSTM-based neural network designed for sequence tagging to additionally generate character-level sequences. The network was jointly trained to produce lemmas, part-of-speech tags and morphological features. Sentence segmentation, tokenization and dependency parsing were handled by UDPipe 1.2 baseline. The results demonstrate the viability of the proposed multitask architecture, although its performance still remains far from state-of-the-art.

Motivation & Objective

• To develop a unified neural architecture that jointly predicts lemmas, part-of-speech tags, and morphological features for words in UD treebanks.
• To extend an existing sequence tagging LSTM model to support character-level sequence generation for lemma prediction.
• To explore the feasibility of multitask learning in morphological and lexical representation learning using shared encoder representations.
• To evaluate the model’s performance on multiple languages in the CoNLL 2018 UD Shared Task, using UDPipe 1.2 as a baseline for parsing and tokenization.
• To identify architectural improvements—such as input injection and positional encoding—for better lemma generation.

Proposed method

• The model uses a bidirectional LSTM encoder to process word sequences, with word embeddings from FastText and character-level embeddings for subword modeling.
• For lemma generation, a separate autoregressive decoder is introduced, conditioned on the final hidden state of the encoder and the position within the word.
• Positional encoding is implemented by injecting the number of remaining characters in the word at each decoding step to improve context awareness.
• The model jointly optimizes three loss functions for POS tagging, morphological features, and lemmatization using weighted sum loss with hyperparameters λp=0.2 for POS to balance task difficulty.
• Dropout is applied before the feature extraction LSTMs to mitigate overfitting, though recurrent dropout and noise injection are not used.
• The system is evaluated on nine UD v2.2 treebanks, with predictions replacing UDPipe 1.2’s outputs for lemma, POS, and features.

Experimental results

Research questions

• RQ1 Can a single neural network jointly predict lemmas, part-of-speech tags, and morphological features with competitive performance?
• RQ2 How does injecting the encoder hidden state and positional information affect lemma generation quality?
• RQ3 What impact do loss weighting coefficients have on balancing the performance of multiple tasks in multitask learning?
• RQ4 Why does the proposed architecture generalize poorly to agglutinative languages like Arabic and Korean despite success on English and European languages?
• RQ5 To what extent does architectural design—such as decoder input design and attention mechanisms—affect the model’s ability to learn morphological patterns?

Key findings

• The model achieved competitive results on English, French, and Swedish treebanks, with lemma F1 scores of 95.84 (EWT), 95.84 (Spoken), and 95.30 (Talbanken), respectively, outperforming UDPipe 1.2 baseline in some cases.
• The addition of the encoder hidden state and position-aware encoding significantly improved lemma generation, especially for English, reducing trivial errors like incorrect suffix removal.
• The model’s performance on Arabic and Korean was notably poor, with lemma F1 scores of 87.41 and 87.03, respectively, indicating limited generalization to agglutinative languages.
• Setting λp=0.2 for POS tagging loss improved lemma generation, suggesting that task loss balancing is critical in multitask training.
• Despite limited hyperparameter tuning, the model demonstrated the viability of joint lemmatization and POS tagging using a shared encoder, laying a foundation for future JointUD systems.