[Paper Review] Multi-Task Learning with Contextualized Word Representations for Extented Named Entity Recognition

Thai-Hoang Pham, Khai Mai+5 more|arXiv (Cornell University)|Feb 26, 2019

Topic ModelingComputer Science31 references3 citations

TL;DR

This paper proposes an end-to-end multi-task learning framework with contextualized word representations (ELMo) for fine-grained named entity recognition (FG-NER), achieving state-of-the-art performance without manual feature engineering or dictionary creation. By jointly training on FG-NER with auxiliary tasks like chunking and POS tagging using hierarchical parameter sharing and unshared neural language models, the model achieves an F1 score of 83.35%, surpassing prior SOTA systems that relied on extensive manual effort.

ABSTRACT

Fine-Grained Named Entity Recognition (FG-NER) is critical for many NLP applications. While classical named entity recognition (NER) has attracted a substantial amount of research, FG-NER is still an open research domain. The current state-of-the-art (SOTA) model for FG-NER relies heavily on manual efforts for building a dictionary and designing hand-crafted features. The end-to-end framework which achieved the SOTA result for NER did not get the competitive result compared to SOTA model for FG-NER. In this paper, we investigate how effective multi-task learning approaches are in an end-to-end framework for FG-NER in different aspects. Our experiments show that using multi-task learning approaches with contextualized word representation can help an end-to-end neural network model achieve SOTA results without using any additional manual effort for creating data and designing features.

Motivation & Objective

• To investigate the effectiveness of multi-task learning in end-to-end neural networks for fine-grained named entity recognition (FG-NER).
• To evaluate different parameter sharing schemes (embedding-shared, RNN-shared, hierarchical-shared) in multi-task sequence labeling for FG-NER.
• To examine the impact of incorporating neural language models (shared vs. unshared) on FG-NER performance.
• To compare contextualized word representations (ELMo) against static embeddings (GloVe) in FG-NER under multi-task learning.
• To achieve state-of-the-art performance in FG-NER without relying on manual feature engineering or external dictionaries.

Proposed method

• The base model uses a CNN-BiLSTM-CRF architecture with word embeddings enhanced by character-level CNNs.
• Multi-task learning is implemented via four parameter sharing schemes: embedding-shared, RNN-shared, hierarchical-shared, and unshared-LM configurations.
• Auxiliary tasks include POS tagging, chunking, and NER (CoNLL and OntoNotes), with hierarchical-sharing allowing task-specific RNNs and shared CRF layers.
• Contextualized word representations are obtained using ELMo, which jointly learns word representations and sentence-level dependencies.
• Neural language models are incorporated either shared across tasks or unshared per task to improve sequence modeling and generalization.
• Model training minimizes negative log-likelihood, and decoding uses the Viterbi algorithm to predict the most likely label sequence.

Experimental results

Research questions

• RQ1 How effective is multi-task learning with different parameter sharing schemes for end-to-end FG-NER?
• RQ2 Does using contextualized word representations (ELMo) improve FG-NER performance compared to static embeddings (GloVe)?
• RQ3 What is the impact of incorporating shared versus unshared neural language models on FG-NER performance?
• RQ4 Can an end-to-end neural network achieve SOTA results in FG-NER without manual feature engineering or dictionary construction?
• RQ5 How does the performance of multi-task learning vary across different auxiliary tasks (e.g., POS, chunking, NER)?

Key findings

• The hierarchical-shared multi-task model with unshared neural language modeling achieved the highest F1 score of 83.35%, surpassing the previous SOTA model (83.14%) that required manual feature engineering.
• Using ELMo instead of GloVe improved the base model F1 score from 81.51% to 82.74%, demonstrating the benefit of contextualized representations for long and complex named entities.
• The hierarchical-sharing scheme outperformed embedding-shared and RNN-shared models, especially when auxiliary tasks were dissimilar to FG-NER (e.g., POS tagging, chunking), indicating better inductive bias.
• Incorporating unshared neural language models yielded better results than shared ones, particularly in multi-task settings, suggesting task-specific language modeling improves generalization.
• The best-performing model achieved SOTA performance without any manual effort for dictionary creation or hand-crafted features, confirming the effectiveness of end-to-end learning with multi-task training.
• The model showed significant improvements on long and complex named entities—e.g., 'Public Institution' improved from 80.00% (GloVe) to 95.00% (ELMo)—highlighting the advantage of contextualized representations.