[Paper Review] Enhancing Few-shot NER with Prompt Ordering based Data Augmentation

Hui‐Ming Wang, Liying Cheng+3 more|arXiv (Cornell University)|May 19, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Prompt Ordering based Data Augmentation (PODA), a novel data augmentation method for few-shot named entity recognition (NER) using autoregressive generative models. By generating diverse, reasonable entity orderings via prompt-based instructions and pairing each with a unique input sequence, PODA avoids the one-to-many mapping problem while improving model generalization and performance across flat and nested NER benchmarks without modifying the backbone model.

ABSTRACT

Recently, data augmentation (DA) methods have been proven to be effective for pre-trained language models (PLMs) in low-resource settings, including few-shot named entity recognition (NER). However, conventional NER DA methods are mostly aimed at sequence labeling models, i.e., token-level classification, and few are compatible with unified autoregressive generation frameworks, which can handle a wider range of NER tasks, such as nested NER. Furthermore, these generation frameworks have a strong assumption that the entities will appear in the target sequence with the same left-to-right order as the source sequence. In this paper, we claim that there is no need to keep this strict order, and more diversified but reasonable target entity sequences can be provided during the training stage as a novel DA method. Nevertheless, a naive mixture of augmented data can confuse the model since one source sequence will then be paired with different target sequences. Therefore, we propose a simple but effective Prompt Ordering based Data Augmentation (PODA) method to improve the training of unified autoregressive generation frameworks under few-shot NER scenarios. Experimental results on three public NER datasets and further analyses demonstrate the effectiveness of our approach.

Motivation & Objective

• To address the limitation of autoregressive generation models in few-shot NER, which assume strict left-to-right entity order in outputs.
• To develop a data augmentation method compatible with unified generative frameworks, especially for nested and complex NER tasks.
• To resolve the one-to-many mapping problem in data augmentation, where a single input could map to multiple valid outputs.
• To improve model generalization and performance in low-resource settings using minimal labeled data.
• To enable effective training of generative PLMs like T5 and BART-NER in few-shot NER without architectural modifications.

Proposed method

• Constructing augmented training samples by generating all possible permutations of entity types as prompt-based order instructions.
• Concatenating each unique order instruction with the input sentence to form a new source sequence, creating one-to-one mappings with re-ordered target sequences.
• Using these paired sequences to train autoregressive models without enforcing left-to-right entity order during training.
• Applying the same prompt instruction ('from left to right') during inference to maintain consistency with standard evaluation.
• Avoiding the multimodality problem by ensuring each source sequence maps to only one target sequence per training example.
• Leveraging the order-agnostic nature of NER to generate semantically valid, diverse target sequences that preserve entity mention and type information.

Experimental results

Research questions

• RQ1 Can data augmentation methods designed for sequence labeling models be effectively adapted to unified autoregressive generation frameworks for NER?
• RQ2 Does relaxing the strict left-to-right entity order assumption in autoregressive NER improve model performance in few-shot settings?
• RQ3 Can prompt-based order instructions resolve the one-to-many mapping problem in data augmentation for generative NER models?
• RQ4 How does the number of generated permutations affect model performance and generalization in few-shot NER?
• RQ5 Can the proposed method generalize across different generative backbones like T5 and BART-NER?

Key findings

• PODA significantly improves F1 scores on CoNLL-2003, ACE-2005, and MIT-Movie datasets across 10-shot, 20-shot, and 50-shot settings, demonstrating consistent gains in few-shot NER.
• The method achieves a 6.07-point F1 improvement over pure T5 on ACE-2005 in the 50-shot setting, showing strong performance gains in low-resource scenarios.
• Using all permutations of entity types leads to better performance than using only one or a few, with Figure 2 showing clear gains as permutation count increases.
• PODA enhances both T5 and Flan-T5, though the improvement is relatively smaller on Flan-T5, possibly due to distributional mismatch between augmented data and instruction-tuned data.
• The method generalizes well to other generative models like BART-NER, improving performance even when the base model performs worse than T5 in low-resource settings.
• Ablation studies confirm that the prompt-based instruction mechanism successfully avoids the one-to-many mapping issue, enabling stable and effective training.