[Paper Review] Weakly Supervised Active Learning with Cluster Annotation

Fábio Perez, Rémi Lebret+1 more|arXiv (Cornell University)|Dec 31, 2018

Machine Learning and Algorithms10 references4 citations

TL;DR

This paper proposes a weakly supervised active learning framework that reduces human annotation effort by allowing experts to label entire clusters of similar images instead of individual samples. By combining uncertainty sampling with cluster annotation, the method achieves test accuracy comparable to fully supervised learning while reducing human interactions by 82% on CIFAR-10 and 87% on EuroSAT.

ABSTRACT

In this work, we introduce a novel framework that employs cluster annotation to boost active learning by reducing the number of human interactions required to train deep neural networks. Instead of annotating single samples individually, humans can also label clusters, producing a higher number of annotated samples with the cost of a small label error. Our experiments show that the proposed framework requires 82% and 87% less human interactions for CIFAR-10 and EuroSAT datasets respectively when compared with the fully-supervised training while maintaining similar performance on the test set.

Motivation & Objective

• Reduce the high cost of human data annotation in deep learning by minimizing the number of individual labeling interactions.
• Address the challenge of scaling active learning to large datasets by leveraging clustering to group similar samples.
• Improve annotation efficiency without sacrificing model performance by allowing experts to label clusters with minor label noise.
• Demonstrate that combining uncertainty sampling with cluster annotation yields better performance and efficiency than either method alone.

Proposed method

• Integrate clustering into the pool-based active learning pipeline, where unlabeled samples are grouped into clusters based on feature embeddings from a pre-trained CNN.
• Present clusters to human experts for labeling, assuming class consistency within clusters, thereby reducing the number of required individual annotations.
• Use a hybrid annotation strategy that alternates or combines uncertainty-based sample selection and cluster labeling to improve model performance.
• Apply a single deep neural network for both feature extraction and classification, avoiding additional training overhead.
• Optimize annotation order by selecting uncertain samples first to clean clusters before cluster labeling, reducing label noise.
• Visualize clusters using representative samples to aid expert annotation, balancing clarity and coverage.

Experimental results

Research questions

• RQ1 Can cluster annotation significantly reduce the number of human interactions required for training deep neural networks?
• RQ2 How does combining uncertainty sampling with cluster annotation affect model performance compared to using either method alone?
• RQ3 What is the impact of annotation order—uncertainty-first versus cluster-first—on label error and model accuracy?
• RQ4 To what extent can cluster annotation maintain performance comparable to fully supervised learning with drastically fewer labeled examples?

Key findings

• The proposed framework reduces human interactions by 82% on CIFAR-10 and 87% on EuroSAT compared to fully supervised training, while achieving similar test accuracy.
• The uncertain+cluster annotation order outperforms cluster+uncertain and other baselines, achieving the highest test accuracy across both datasets.
• Label error for cluster-annotated samples is significantly reduced when uncertain samples are annotated first, indicating improved cluster quality.
• Scenarios combining uncertainty and cluster annotation yield the highest total number of annotated images, though performance gains are limited by higher label noise.
• The framework achieves test accuracy of 0.950±0.003 on CIFAR-10 and 0.973±0.001 on EuroSAT with only 9,000 (18%) and 2,800 (13%) human interactions, respectively, matching fully supervised performance.
• Cluster-only strategies stagnate early, highlighting the need for continued uncertainty-based sampling to maintain performance gains.