[Paper Review] Adaptive Self-training for Few-shot Neural Sequence Labeling

Yaqing Wang, Subhabrata Mukherjee+5 more|arXiv (Cornell University)|Oct 7, 2020

Topic Modeling46 references35 citations

TL;DR

The paper presents MetaST, a framework combining adaptive self-training with token-level meta-learning to re-weight pseudo-labeled data, enabling neural sequence labeling with very few labeled examples and abundant unlabeled data.

ABSTRACT

Sequence labeling is an important technique employed for many Natural Language Processing (NLP) tasks, such as Named Entity Recognition (NER), slot tagging for dialog systems and semantic parsing. Large-scale pre-trained language models obtain very good performance on these tasks when fine-tuned on large amounts of task-specific labeled data. However, such large-scale labeled datasets are difficult to obtain for several tasks and domains due to the high cost of human annotation as well as privacy and data access constraints for sensitive user applications. This is exacerbated for sequence labeling tasks requiring such annotations at token-level. In this work, we develop techniques to address the label scarcity challenge for neural sequence labeling models. Specifically, we develop self-training and meta-learning techniques for training neural sequence taggers with few labels. While self-training serves as an effective mechanism to learn from large amounts of unlabeled data -- meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels. Extensive experiments on six benchmark datasets including two for massive multilingual NER and four slot tagging datasets for task-oriented dialog systems demonstrate the effectiveness of our method. With only 10 labeled examples for each class for each task, our method obtains 10% improvement over state-of-the-art systems demonstrating its effectiveness for the low-resource setting.

Motivation & Objective

• Address label scarcity in neural sequence labeling (NER and slot tagging) by leveraging unlabeled data.
• Develop an end-to-end framework that adaptively selects informative labeled validation data.
• Mitigate error propagation from noisy pseudo-labels via meta-learning-based token-level re-weighting.
• Demonstrate generalization across multilingual NER and task-oriented dialog datasets under few-shot settings.

Proposed method

• Use a pre-trained language model (teacher) fine-tuned on small labeled data to generate pseudo-labels for unlabeled data.
• Adaptive labeled data acquisition to create a held-out validation set on the fly using loss decay as a proxy for uncertainty.
• Meta-learning to re-weight pseudo-labeled tokens based on the student’s loss on the adaptive validation set.
• Token-level re-weighting of pseudo-labeled data via gradient-based perturbations, aggregated over multiple labeled batches to obtain robust weights.
• End-to-end training where the student learns from weighted pseudo-labels, and the teacher is updated iteratively with the student’s parameters.
• Compare MetaST against fully supervised BERT and several semi-supervised baselines across six datasets.

Experimental results

Research questions

• RQ1 Can adaptive self-training with token-level meta-learning improve few-shot sequence labeling performance across diverse datasets and languages?
• RQ2 Does adaptive validation set construction and token-level re-weighting effectively mitigate noise from pseudo-labels in sequence tagging?
• RQ3 How does MetaST perform relative to strong baselines under varying amounts of labeled data and unlabeled data?
• RQ4 Is iterative teacher-student updating beneficial for maintaining task focus during self-training?
• RQ5 Do components like adaptive data acquisition and token-level re-weighting contribute significantly to overall gains?

Key findings

• MetaST outperforms all baselines on six datasets with 10 labeled examples per slot, achieving average improvements of 8.82–18.07 percentage points over baselines using the same encoder.
• MetaST narrows the gap to fully supervised BERT in some datasets (e.g., SNIPS and Email) under few-shot settings.
• Token-level re-weighting via meta-learning significantly boosts performance, while soft pseudo-labels hurt compared to hard labels.
• Adaptive labeled data acquisition improves performance, especially for tasks with many slots and high diversity (e.g., SNIPS).
• Continued in-domain pre-training helps some tasks but MetaST consistently provides gains across both in-domain and cross-domain datasets.