[Paper Review] VTCC-NLP at NL4Opt competition subtask 1: An Ensemble Pre-trained language models for Named Entity Recognition

Xuan-Dung Doan|arXiv (Cornell University)|Dec 14, 2022

Topic Modeling4 citations

TL;DR

This paper presents an ensemble of three pre-trained language models—XLM-R, BART, and DeBERTa-V3—combined via averaging contextual embeddings to improve Named Entity Recognition (NER) performance on the NL4Opt competition subtask 1. The method achieves a 92.9% F1 score on the test set, ranking 5th, by leveraging diverse contextual representations and a CRF layer for sequence labeling.

ABSTRACT

We propose a combined three pre-trained language models (XLM-R, BART, and DeBERTa-V3) as an empower of contextualized embedding for named entity recognition. Our model achieves a 92.9% F1 score on the test set and ranks 5th on the leaderboard at NL4Opt competition subtask 1.

Motivation & Objective

• To improve Named Entity Recognition (NER) performance on low-resource and cross-domain text using pre-trained language models.
• To investigate the effectiveness of combining multiple pre-trained models for enhanced contextual representation in NER.
• To explore whether ensemble methods outperform individual models in flat NER tasks across diverse domains.
• To evaluate the impact of auxiliary components like ELMo and GCN on the ensemble model's performance.
• To achieve state-of-the-art results on the NL4Opt competition subtask 1 test set using a robust, multi-model ensemble approach.

Proposed method

• The model uses three pre-trained language models—XLM-R, BART, and DeBERTa-V3—each processing the same input sentence to generate contextualized embeddings.
• Subword tokenization is selected per model, and the representation with the minimum sequence length is used to ensure consistency across models.
• Contextual embeddings from the three models are averaged to form a combined representation: $ V = \frac{1}{3}(V_1 + V_2 + V_3) $, where $ V_1, V_2, V_3 $ are the outputs from BART, XLM-R, and DeBERTa-V3 respectively.
• The averaged embeddings are passed through a linear-chain Conditional Random Field (CRF) layer to predict BIO-tagged sequences with structured prediction.
• The CRF layer computes the conditional probability $ p_{\theta}(y|\hat{x}) $ using transition scores and emission scores, optimizing via negative log-likelihood loss.
• Model selection is based on the best F1 score on the development set, with checkpoints saved every epoch and the top-performing model used for inference.

Experimental results

Research questions

• RQ1 Can combining multiple pre-trained language models via embedding averaging improve NER performance on cross-domain and low-resource text?
• RQ2 How does the ensemble of XLM-R, BART, and DeBERTa-V3 compare to individual models in terms of F1 score and robustness across domains?
• RQ3 Does the integration of ELMo or GCN-based features enhance the performance of the ensemble model on the NL4Opt dataset?
• RQ4 To what extent does the ensemble method maintain or improve performance when tested on unseen domains (e.g., transportation, production, science)?
• RQ5 Is the CRF layer effective in refining the sequence-level predictions when combined with averaged contextual embeddings from diverse pre-trained models?

Key findings

• The ensemble model, combining XLM-R, BART, and DeBERTa-V3 via embedding averaging, achieved a 92.9% F1 score on the NL4Opt competition test set.
• The ensemble model outperformed all individual models on the development set, demonstrating the benefit of combining diverse pre-trained representations.
• The use of ELMo improved the baseline model’s performance but led to a drop in accuracy when applied to the ensemble model, indicating potential interference.
• GCN integration, inspired by dependency parsing, similarly improved the baseline but degraded the performance of the ensemble model, suggesting incompatibility with the ensemble strategy.
• The final model ranked 5th on the NL4Opt competition leaderboard, confirming the effectiveness of the ensemble approach in cross-domain NER tasks.
• The model achieved optimal performance after 50 training epochs with a learning rate of 1e-4 and gradient accumulation of 4, using a single 40GB A100 GPU.