[Paper Review] Discriminative Active Learning

Daniel Gissin, Shai Shalev‐Shwartz|arXiv (Cornell University)|Jul 15, 2019

Machine Learning and AlgorithmsComputer Science27 references109 citations

TL;DR

DAL reframes batch active learning as a binary classification task to make labeled and unlabeled data indistinguishable, enabling efficient selection of informative batches for neural networks.

ABSTRACT

We propose a new batch mode active learning algorithm designed for neural networks and large query batch sizes. The method, Discriminative Active Learning (DAL), poses active learning as a binary classification task, attempting to choose examples to label in such a way as to make the labeled set and the unlabeled pool indistinguishable. Experimenting on image classification tasks, we empirically show our method to be on par with state of the art methods in medium and large query batch sizes, while being simple to implement and also extend to other domains besides classification tasks. Our experiments also show that none of the state of the art methods of today are clearly better than uncertainty sampling when the batch size is relatively large, negating some of the reported results in the recent literature.

Motivation & Objective

• Motivate reducing labeling cost in deep learning by selecting informative labeled data.
• Propose a batch-mode active learning method that is task-agnostic and relies on learned representations.
• Show that making labeled and unlabeled sets indistinguishable improves batch selection.
• Demonstrate competitiveness with state-of-the-art methods on image classification tasks.
• Discuss practical considerations and potential extensions beyond classification tasks.

Proposed method

• Formulate active learning as a binary classification problem distinguishing labeled vs. unlabeled data in a learned representation space Ψ(x).
• Train a binary classifier on U ∪ L to approximate P(y|Ψ(x)) for y ∈ {l, u}.
• Select top-K unlabeled examples with the highest posterior P(y=u|Ψ(x)).
• Use batch-wise mini-queries to trade off speed and diversity without changing Ψ between mini-queries.
• Choose a simple MLP (three hidden layers, width 256) for the binary classifier to fit the training samples.
• Optionally limit training to ~98% accuracy to ensure informative uncertain examples are identified.

Experimental results

Research questions

• RQ1 Can active learning be effectively performed by treating the problem as a binary discrimination task between labeled and unlabeled data?
• RQ2 How does the Discriminative Active Learning (DAL) approach compare to uncertainty-based and core-set methods across varying batch sizes?
• RQ3 Does using a learned representation Ψ(x) improve performance over using the raw input space for batch selection?
• RQ4 Can DAL extend beyond classification tasks to other domains that use learned representations?
• RQ5 What practical considerations affect the speed-diversity trade-off in batch querying?

Key findings

• DAL achieves competitive accuracy with state-of-the-art methods on MNIST and CIFAR-10 for medium to large batch sizes.
• Uncertainty-based methods perform well at small batch sizes, but all methods converge in performance at very large batch sizes.
• DAL ranks unlabeled data differently from uncertainty and margin-based methods, indicating a distinct selection strategy.
• Core-Set performs well only at very small batch sizes; DAL often surpasses Core-Set in small-batch scenarios.
• Batch querying with DAL can be made more diverse via mini-queries without sacrificing performance.