[Paper Review] Feature Binding with Category-Dependant MixUp for Semantic Segmentation and Adversarial Robustness

Amirul Islam, Matthew Kowal+2 more|arXiv (Cornell University)|Aug 13, 2020

Adversarial Robustness in Machine Learning38 references4 citations

TL;DR

This paper proposes category-dependent MixUp for semantic segmentation, where images are blended based on class labels to train a network that simultaneously segments and separates source images. The method improves mIoU by 2.1% over DeepLabv3 and enhances robustness against adversarial attacks, especially under extreme perturbations, by enforcing feature binding through structured data mixing and denoising.

ABSTRACT

In this paper, we present a strategy for training convolutional neural networks to effectively resolve interference arising from competing hypotheses relating to inter-categorical information throughout the network. The premise is based on the notion of feature binding, which is defined as the process by which activation's spread across space and layers in the network are successfully integrated to arrive at a correct inference decision. In our work, this is accomplished for the task of dense image labelling by blending images based on their class labels, and then training a feature binding network, which simultaneously segments and separates the blended images. Subsequent feature denoising to suppress noisy activations reveals additional desirable properties and high degrees of successful predictions. Through this process, we reveal a general mechanism, distinct from any prior methods, for boosting the performance of the base segmentation network while simultaneously increasing robustness to adversarial attacks.

Motivation & Objective

• To address interference from competing hypotheses in semantic segmentation by improving feature binding across categories.
• To enhance model robustness against adversarial attacks without architectural changes.
• To develop a training strategy that simultaneously performs dense image labeling and blind source separation.
• To reduce reliance on architectural innovation by leveraging structured data augmentation for improved generalization.
• To investigate whether category-aware image blending can yield better contextual understanding and prediction accuracy.

Proposed method

• Images are blended using a category-dependent strategy that clusters samples by class to create structured 'categorical collisions' in the training data.
• A feature binding network is trained on the blended dataset to simultaneously segment and separate the source images, enforcing correspondence between spatial and feature activations.
• A feature denoising stage is applied post-training to suppress noisy activations and refine predictions.
• A feature binding head is introduced to guide the network in making informed final predictions based on source and phantom activations.
• The method uses a ResNet101-based distributed gating network as the backbone and applies fine-tuning with denoising to improve performance.
• The training process enforces that the network learns to resolve ambiguous activations by associating features from different sources to their correct semantic regions.

Experimental results

Research questions

• RQ1 Can structured, category-dependent image blending improve semantic segmentation performance beyond standard MixUp and CutMix?
• RQ2 Does enforcing feature binding during training enhance robustness to adversarial attacks?
• RQ3 How does the choice of image pairing strategy (e.g., random vs. category-based) affect model generalization and performance?
• RQ4 To what extent does feature denoising improve prediction accuracy and robustness in the presence of adversarial perturbations?
• RQ5 Can a single network architecture simultaneously perform dense labeling and blind source separation through data-level augmentation?

Key findings

• The proposed category-dependent MixUp method achieves a mean Intersection over Union (mIoU) of 78.0 on the PASCAL VOC 2012 validation set, a 2.1% improvement over DeepLabv3 (75.9).
• The method significantly outperforms standard MixUp and CutMix, with CutMix’s mIoU dropping under adversarial attacks while the proposed method remains robust.
• Feature denoising increases mIoU from 76.2 to 78.0 when applied after training, demonstrating its effectiveness in refining predictions.
• The feature binding head improves performance from 75.3 to 76.1 mIoU, indicating its role in enabling better final predictions through refined activation integration.
• The model exhibits strong robustness under extreme adversarial settings (R-No and R-Part), outperforming DeepLabv3, Mixup, and CutMix in adversarial generalization.
• Clustering-based blending (C_fb) yields higher mIoU than random pairing (R_fb) or within-category mixing (Ca_fb), proving the importance of strategic collision design in data augmentation.