[論文レビュー] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

Sungmin Cha, beomyoung kim+2 more|arXiv (Cornell University)|Jun 22, 2021

Domain Adaptation and Few-Shot Learning被引用数 26

ひとこと要約

SSUL-M は 背景ピクセル用の Unknown ラベルを導入し、ラベル拡張、モデル凍結とシグモイド BCE による安定したスコア学習、そして tiny exemplar memory を用いて exemplar-based class-incremental semantic segmentation (CISS) を前進させる。

ABSTRACT

This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.

研究の動機と目的

• catastrophic forgetting and background semantic drift in class-incremental semantic segmentation (CISS).
• Propose a robust baseline (SSUL-M) that improves plasticity and stability via label augmentation, stable score learning, and exemplar-memory integration.
• Show that tiny exemplar memory improves both past and new class learning in CISS.

提案手法

• Define an Unknown background class label c_u to accompany the standard background c_b within the BG region.
• Augment training targets with pseudo-labels from the previous model and with the Unknown class using saliency maps to distinguish future-class potential regions from true background.
• Use separate sigmoid (binary cross-entropy) outputs per class with model freezing of the backbone and past classifiers to stabilize scores across incremental steps.
• Initialize new task classifiers for current classes using the previous task's Unknown class classifier to promote fast and stable learning.
• Maintain a tiny, class-balanced exemplar memory M of past data to anchor learning and improve prediction precision for both past and current classes.

実験結果

リサーチクエスチョン

• RQ1 Can introducing an Unknown label for BG regions improve plasticity to accommodate future classes in CISS?
• RQ2 Does freezing the backbone and using sigmoid BCE with targeted initialization provide greater stability than softmax cross-entropy in CISS?
• RQ3 Can a tiny, class-balanced exemplar memory improve both plasticity and stability in exemplar-based CISS without sacrificing efficiency?

主な発見

• SSUL-M achieves state-of-the-art performance on Pascal VOC 2012 and ADE20K across multiple incremental setups.
• The Unknown BG label, together with pseudo-labeling, improves both plasticity and stability, outperforming baselines that do not use an Unknown class or exemplar memory.
• Model freezing with sigmoid BCE provides more stable per-class scores than softmax CE in the incremental setting.
• Exemplar memory (SSUL-M) further boosts mIoU, especially for newly learned classes, with class-balanced sampling outperforming random sampling.
• Weight transfer from the Unknown class classifier accelerates and stabilizes learning of new classes.