[Paper Review] AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER

Weile Chen, Huiqiang Jiang+3 more|arXiv (Cornell University)|Jun 4, 2021

Topic Modeling35 references4 citations

TL;DR

AdvPicker proposes a novel adversarial data selection framework for cross-lingual NER that leverages unlabeled target-language data by training a discriminator to identify language-independent, high-quality pseudo-labeled examples. By combining adversarial training with knowledge distillation, it outperforms state-of-the-art methods without requiring external resources like translation models or gazetteers.

ABSTRACT

Neural methods have been shown to achieve high performance in Named Entity Recognition (NER), but rely on costly high-quality labeled data for training, which is not always available across languages. While previous works have shown that unlabeled data in a target language can be used to improve cross-lingual model performance, we propose a novel adversarial approach (AdvPicker) to better leverage such data and further improve results. We design an adversarial learning framework in which an encoder learns entity domain knowledge from labeled source-language data and better shared features are captured via adversarial training - where a discriminator selects less language-dependent target-language data via similarity to the source language. Experimental results on standard benchmark datasets well demonstrate that the proposed method benefits strongly from this data selection process and outperforms existing state-of-the-art methods; without requiring any additional external resources (e.g., gazetteers or via machine translation). The code is available at https://aka.ms/AdvPicker

Motivation & Objective

• Address the challenge of low-resource named entity recognition in low-resource languages by leveraging unlabeled target-language data.
• Overcome limitations of existing pseudo-labeling methods that assume all pseudo-labeled data is beneficial, which can introduce noise and language-specific bias.
• Improve cross-lingual transfer performance by selecting only language-independent, high-quality pseudo-labeled examples through adversarial learning.
• Develop a method that combines feature-based and pseudo-labeling approaches without relying on costly external resources such as machine translation or parallel corpora.

Proposed method

• Train a source-language NER model (mBERT) with token-level adversarial training to learn language-invariant representations.
• Train a language discriminator to distinguish between source-language and target-language token-level representations, encouraging the encoder to minimize language-specific features.
• Use the pre-trained encoder to generate pseudo-labels on unlabeled target-language data.
• Apply the trained discriminator to select only the most language-independent (i.e., source-like) pseudo-labeled examples from the target language.
• Perform knowledge distillation to train a final target-language NER model on the selected, high-quality pseudo-labeled data.
• Use a soft-label distillation strategy where the student model is trained on predictions from multiple teacher models with different random seeds to improve robustness.

Experimental results

Research questions

• RQ1 Can adversarial training effectively identify language-independent, high-quality pseudo-labeled examples from unlabeled target-language data in cross-lingual NER?
• RQ2 Does selecting only language-independent pseudo-labeled data improve model generalization and performance compared to using all pseudo-labeled data?
• RQ3 Can the proposed method achieve state-of-the-art performance in zero-shot cross-lingual NER without relying on external resources like translation models or gazetteers?
• RQ4 How does the adversarial data selection mechanism affect model stability and robustness across multiple random seeds?

Key findings

• AdvPicker achieves new state-of-the-art F1 scores on three target languages (German, Spanish, Dutch) across standard benchmark datasets, with gains of up to 1.5–2.0 F1 points over previous SOTA methods.
• The average F1 score for language-independent data is 12.87 points higher than for language-specific data, confirming the quality of the selected examples.
• Even when evaluated on the 'Other' (non-selected) data split, AdvPicker outperforms all baselines, indicating it generalizes well beyond the selected data distribution.
• The ablation study shows that knowledge distillation on selected data improves performance significantly, and using all pseudo-labeled data leads to performance degradation, validating the importance of selective data usage.
• AdvPicker exhibits lower standard deviation in F1 scores across five random seeds than mBERT-ft and UniTrans, demonstrating improved training stability.
• The method achieves consistent improvements across all target languages, with larger gains on larger datasets (e.g., German and Dutch), suggesting scalability to higher-resource target languages.