[Paper Review] Exemplar Guided Active Learning

Jason Hartford, Kevin Leyton‐Brown+4 more|arXiv (Cornell University)|Nov 2, 2020

Natural Language Processing Techniques20 references4 citations

TL;DR

This paper proposes Exemplar Guided Active Learning (EGAL), a method that uses contextual embeddings from pre-trained language models to efficiently identify rare word senses in highly skewed datasets, while employing a statistical stopping rule to halt labeling for senses below a frequency threshold. The approach reduces labeling cost by up to 50% compared to standard active learning, with theoretical guarantees of logarithmic overhead over an oracle that knows true label frequencies.

ABSTRACT

We consider the problem of wisely using a limited budget to label a small subset of a large unlabeled dataset. We are motivated by the NLP problem of word sense disambiguation. For any word, we have a set of candidate labels from a knowledge base, but the label set is not necessarily representative of what occurs in the data: there may exist labels in the knowledge base that very rarely occur in the corpus because the sense is rare in modern English; and conversely there may exist true labels that do not exist in our knowledge base. Our aim is to obtain a classifier that performs as well as possible on examples of each "common class" that occurs with frequency above a given threshold in the unlabeled set while annotating as few examples as possible from "rare classes" whose labels occur with less than this frequency. The challenge is that we are not informed which labels are common and which are rare, and the true label distribution may exhibit extreme skew. We describe an active learning approach that (1) explicitly searches for rare classes by leveraging the contextual embedding spaces provided by modern language models, and (2) incorporates a stopping rule that ignores classes once we prove that they occur below our target threshold with high probability. We prove that our algorithm only costs logarithmically more than a hypothetical approach that knows all true label frequencies and show experimentally that incorporating automated search can significantly reduce the number of samples needed to reach target accuracy levels.

Motivation & Objective

• To address the challenge of labeling word sense disambiguation datasets under extreme class imbalance, where rare senses are underrepresented and standard active learning fails.
• To develop a method that identifies rare word senses without prior knowledge of their true frequencies, using contextual embeddings from pre-trained language models.
• To design a stopping rule that proves with high probability that a sense is below a target frequency threshold, thus avoiding unnecessary labeling.
• To maintain balanced labeling for common senses while minimizing annotation effort on rare or non-existent senses.
• To evaluate the method on a new, highly skewed Reddit-based word sense disambiguation dataset.

Proposed method

• Leverages pre-trained language model embeddings to find exemplars of each candidate sense in a word's context, enabling localized search in embedding space.
• Uses a frequency threshold γ to define 'common' vs. 'rare' senses, with common senses requiring balanced labeling.
• Employs a statistical stopping rule based on importance-weighted sampling to prove with high confidence that a sense's true frequency is below γ.
• Switches from guided search to uncertainty-based active learning once common senses are identified and labeled.
• Automates the guided learning process by using embeddings to search for examples of each sense, eliminating manual search by annotators.
• Estimates label frequency ˆpy online using importance-weighted samples from the embedding neighborhood, enabling real-time stopping decisions.

Experimental results

Research questions

• RQ1 Can we efficiently discover rare word senses in a large unlabeled corpus without prior knowledge of their true frequencies?
• RQ2 How can we reduce labeling cost in highly skewed datasets while ensuring balanced coverage of common senses?
• RQ3 What stopping rule can reliably identify and exclude rare senses with minimal labeling, even when the candidate label set is misspecified?
• RQ4 How does exemplar-based search in embedding space compare to manual search in terms of sample efficiency and accuracy?
• RQ5 To what extent does the proposed method reduce labeling cost compared to standard active learning under extreme class imbalance?

Key findings

• The EGAL algorithm reduces labeling cost by up to 50% compared to standard active learning on highly skewed word sense disambiguation tasks.
• The stopping rule successfully identifies and excludes rare senses with high confidence, reducing unnecessary labeling by avoiding false positives in rare class discovery.
• The method achieves near-optimal performance with only logarithmic overhead compared to an oracle that knows all true label frequencies.
• On the new Reddit-based word sense disambiguation dataset, EGAL achieves higher average class coverage for common senses at lower labeling budgets than competing methods.
• The use of contextual embeddings enables effective search for rare senses even when they are orders of magnitude less frequent than common ones.
• The hybrid approach—exemplar-guided search followed by uncertainty-based active learning—outperforms pure active learning or guided learning in extreme skew scenarios.