[Paper Review] Few-shot Text Classification with Distributional Signatures

Yujia Bao, Menghua Wu+2 more|arXiv (Cornell University)|Aug 16, 2019

Domain Adaptation and Few-Shot LearningComputer Science44 references88 citations

TL;DR

Proposes a meta-learning model that uses distributional signatures to generate word-level attention, enabling effective few-shot text and relation classification, outperforming lexical baselines.

ABSTRACT

In this paper, we explore meta-learning for few-shot text classification. Meta-learning has shown strong performance in computer vision, where low-level patterns are transferable across learning tasks. However, directly applying this approach to text is challenging--lexical features highly informative for one task may be insignificant for another. Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns. Our model is trained within a meta-learning framework to map these signatures into attention scores, which are then used to weight the lexical representations of words. We demonstrate that our model consistently outperforms prototypical networks learned on lexical knowledge (Snell et al., 2017) in both few-shot text classification and relation classification by a significant margin across six benchmark datasets (20.0% on average in 1-shot classification).

Motivation & Objective

• Motivate and improve few-shot text classification where lexical features alone transfer poorly across tasks.
• Leverage distributional signatures (e.g., unigram statistics) to transfer attention across tasks.
• Integrate an attention generator with a ridge regressor to enable rapid adaptation from few labeled examples.
• Provide theoretical justification of robustness to word-substitution perturbations.
• Empirically validate on six datasets for text and relation classification, showing gains over baselines.

Proposed method

• Define distributional signatures s(x) and t(x) to capture general word importance and class-specific importance.
• Concatenate s(x) and t(x) and fuse with a biLSTM to compute word-level attention α_i via a dot-product with a learned vector.
• Use an attention-weighted lexical representation φ(x) = sum_i α_i f_ebd(x_i) for each input word embedding.
• Train a ridge regressor on the support set to map φ(x) to class labels with a closed-form solution W = Φ_S^T(Φ_SΦ_S^T + λI)^{-1}Y_S.
• Calibrate predictions on the query set with Y_Q_hat = a Φ_Q W + b and apply softmax for probabilities.
• Train the attention generator end-to-end via the cross-entropy loss on the query set, using the ridge regressor feedback.
• Prove that AttGen is invariant to word-substitution perturbations under a bijection that preserves unigram probabilities.

Experimental results

Research questions

• RQ1 Can distributional signatures be used to transfer attention across NLP tasks for robust few-shot learning?
• RQ2 Does an attention generator guided by distributional signatures improve over lexical meta-learners on text and relation classification in few-shot settings?
• RQ3 Is the learned attention robust to word-substitution perturbations and generalize to unseen classes?
• RQ4 How does the proposed architecture perform across diverse datasets (text and relation), in 1-shot and 5-shot regimes?

Key findings

• The proposed model achieves the best accuracy across six datasets in both 1-shot and 5-shot settings.
• On average, the model improves 5-way 1-shot accuracy by 7.5% and 5-way 5-shot accuracy by 3.9% over the best baseline.
• Compared with cnn + proto, the model improves by 20.0% on average in 1-shot classification on Reuters/other datasets.
• Ablations show both s(·) and t(·) contribute to performance, with s(·) having a larger impact, and biLSTM fusion improves results slightly.
• The model generalizes better to unseen classes than lexicon-aware meta-learners, as illustrated by learning curves on Reuters and qualitative attention visualizations.
• BERT contextualization boosts FewRel performance but not HuffPost, indicating task-dependent gains from contextualized representations.