[Paper Review] Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation

Yukun Su, Ruizhou Sun+2 more|arXiv (Cornell University)|Mar 2, 2021

Advanced Neural Network Applications49 references9 citations

TL;DR

This paper proposes Context Decoupling Augmentation (CDA), a novel data augmentation method for weakly supervised semantic segmentation (WSSS) that randomly pastes object instances into novel backgrounds to break object-context dependencies. By training with these decoupled samples, models learn to focus on intrinsic object features rather than spurious contextual cues, achieving new state-of-the-art performance with 66.8% mIoU on PASCAL VOC 2012 test set and 33.7% on COCO val set.

ABSTRACT

Data augmentation is vital for deep learning neural networks. By providing massive training samples, it helps to improve the generalization ability of the model. Weakly supervised semantic segmentation (WSSS) is a challenging problem that has been deeply studied in recent years, conventional data augmentation approaches for WSSS usually employ geometrical transformations, random cropping and color jittering. However, merely increasing the same contextual semantic data does not bring much gain to the networks to distinguish the objects, e.g., the correct image-level classification of "aeroplane" may be not only due to the recognition of the object itself, but also its co-occurrence context like "sky", which will cause the model to focus less on the object features. To this end, we present a Context Decoupling Augmentation (CDA) method, to change the inherent context in which the objects appear and thus drive the network to remove the dependence between object instances and contextual information. To validate the effectiveness of the proposed method, extensive experiments on PASCAL VOC 2012 dataset with several alternative network architectures demonstrate that CDA can boost various popular WSSS methods to the new state-of-the-art by a large margin.

Motivation & Objective

• To address the issue of deep networks relying on spurious contextual cues (e.g., 'sky' for 'aeroplane') rather than intrinsic object features in weakly supervised semantic segmentation.
• To reduce the model's dependence on co-occurring background contexts that hinder accurate object localization.
• To develop a data augmentation strategy that enhances feature learning without requiring additional annotations or object-level labels.
• To achieve state-of-the-art performance on PASCAL VOC 2012 and COCO benchmarks using only image-level labels.

Proposed method

• CDA uses a two-stage training process where object instances are first predicted via off-the-shelf WSSS models.
• Selected foreground instances are randomly pasted into different background scenes of the original images to create augmented training samples.
• The augmentation is performed online, with random combinations of objects and backgrounds in each training iteration to maximize diversity.
• The model is trained on both original and augmented images, forcing it to learn discriminative object features independent of context.
• The method is compatible with various WSSS architectures and does not require additional annotations or model retraining.
• A pairwise training strategy is employed to improve the classifier’s ability to distinguish object-specific features.

Experimental results

Research questions

• RQ1 Can decoupling object instances from their original contextual backgrounds improve feature learning in weakly supervised semantic segmentation?
• RQ2 Does random re-pasting of object instances into novel scenes reduce reliance on spurious contextual cues?
• RQ3 Can a context-decoupling data augmentation strategy boost performance across multiple WSSS baselines without additional annotations?
• RQ4 To what extent does CDA improve mIoU on standard benchmarks like PASCAL VOC 2012 and COCO?

Key findings

• CDA achieves 66.8% mIoU on the PASCAL VOC 2012 test set, setting a new state-of-the-art.
• On the COCO validation set, CDA boosts performance to 33.7% mIoU, surpassing the previous best by 1.1%.
• The method improves all tested baselines, including IRNet and SEAM, by more than 2.8% mIoU on average.
• Qualitative results show that CDA produces more accurate and complete object masks with better boundary delineation.
• SEAM + CDA achieves the best performance on both PASCAL VOC 2012 validation and test sets, outperforming prior state-of-the-art methods.
• The ablation study confirms that CDA effectively reduces model dependence on contextual cues, leading to improved generalization.