[Paper Review] Continual Semi-Supervised Learning through Contrastive Interpolation Consistency

Matteo Boschini, Pietro Buzzega+3 more|arXiv (Cornell University)|Aug 14, 2021

Domain Adaptation and Few-Shot LearningComputer Science38 references32 citations

TL;DR

This paper proposes Contrastive Continual Interpolation Consistency (CCIC), a novel method for Continual Semi-Supervised Learning (CSSL) that combines rehearsal with contrastive consistency regularization to leverage unlabeled data. Surprisingly, CCIC achieves performance surpassing SOTA fully supervised methods using only 25% labeled data, demonstrating that limited supervision with effective semi-supervised regularization can outperform full supervision in continual learning settings.

ABSTRACT

Continual Learning (CL) investigates how to train Deep Networks on a stream of tasks without incurring forgetting. CL settings proposed in literature assume that every incoming example is paired with ground-truth annotations. However, this clashes with many real-world applications: gathering labeled data, which is in itself tedious and expensive, becomes infeasible when data flow as a stream. This work explores Continual Semi-Supervised Learning (CSSL): here, only a small fraction of labeled input examples are shown to the learner. We assess how current CL methods (e.g.: EWC, LwF, iCaRL, ER, GDumb, DER) perform in this novel and challenging scenario, where overfitting entangles forgetting. Subsequently, we design a novel CSSL method that exploits metric learning and consistency regularization to leverage unlabeled examples while learning. We show that our proposal exhibits higher resilience to diminishing supervision and, even more surprisingly, relying only on 25% supervision suffices to outperform SOTA methods trained under full supervision.

Motivation & Objective

• To address the challenge of catastrophic forgetting in continual learning when only a small fraction of data is labeled.
• To explore whether semi-supervised learning techniques can effectively improve performance under low supervision in continual learning.
• To design a new method that integrates supervised and unsupervised learning in a unified, end-to-end manner during incremental task learning.
• To demonstrate that limited labeling can be more effective than full supervision when combined with proper consistency regularization.

Proposed method

• CCIC combines experience replay with contrastive consistency regularization, using memory buffers to store past examples.
• It enforces consistency between augmented views of the same input via contrastive loss, promoting robust feature learning.
• It introduces interpolation consistency, where interpolated representations between labeled and unlabeled examples are regularized to be close in feature space.
• The method uses across-task mining to select negative samples from previous tasks, pushing their features away from current samples.
• It incorporates a temporal consistency baseline using an exponential moving average of model weights, though this is outperformed by other components.
• The framework is trained end-to-end with a combination of supervised cross-entropy loss, contrastive loss, and interpolation consistency loss.

Experimental results

Research questions

• RQ1 Can semi-supervised learning techniques improve continual learning performance under low supervision?
• RQ2 Does a unified training scheme that interleaves labeled and unlabeled data outperform separate training phases?
• RQ3 Can consistency regularization via contrastive learning mitigate forgetting in continual learning with limited labels?
• RQ4 Is 25% supervision sufficient to surpass SOTA methods trained with full supervision in continual learning?
• RQ5 How does the choice of negative mining strategy impact performance in the CSSL setting?

Key findings

• CCIC achieves 84.7% average accuracy on CIFAR-100 with 25% labels and a buffer size of 5120, outperforming SOTA fully supervised methods.
• With only 0.8% labels, CCIC maintains 55.2% accuracy on CIFAR-100, while ER drops to 25.9%, showing superior robustness to label scarcity.
• On SVHN, CCIC with 5120 buffer size achieves 84.7% accuracy at 25% supervision, surpassing the SOTA fully supervised result.
• The performance drop from 25% to 0.8% labels is only 26% on average for CCIC, compared to 47% for ER, indicating greater resilience to diminishing supervision.
• Task-agnostic and within-task mining strategies underperform compared to across-task mining, confirming the importance of task-aware negative sampling.
• The use of exponential moving average for consistency regularization underperforms all other components, suggesting it is less effective in the continual learning setting.