[Paper Review] VisualBackProp for learning using privileged information with CNNs

Devansh Bisla, Anna Choromanska|arXiv (Cornell University)|May 24, 2018

Machine Learning and Data Classification21 references3 citations

TL;DR

This paper proposes VisualBackProp, a method to enhance CNN training by incorporating privileged information—such as segmentation masks—through a lightweight, parameter-free architectural module that guides attention to relevant image regions during training. The approach improves classification accuracy by 2.4% on ImageNet and 2.7% on PASCAL VOC by enforcing attention on salient features via reverse visualization, without modifying the inference pipeline.

ABSTRACT

In many machine learning applications, from medical diagnostics to autonomous driving, the availability of prior knowledge can be used to improve the predictive performance of learning algorithms and incorporate `physical,' `domain knowledge,' or `common sense' concepts into training of machine learning systems as well as verify constraints/properties of the systems. We explore the learning using privileged information paradigm and show how to incorporate the privileged information, such as segmentation mask available along with the classification label of each example, into the training stage of convolutional neural networks. This is done by augmenting the CNN model with an architectural component that effectively focuses model's attention on the desired region of the input image during the training process and that is transparent to the network's label prediction mechanism at testing. This component effectively corresponds to the visualization strategy for identifying the parts of the input, often referred to as visualization mask, that most contribute to the prediction, yet uses this strategy in reverse to the classical setting in order to enforce the desired visualization mask instead. We verify our proposed algorithms through exhaustive experiments on benchmark ImageNet and PASCAL VOC data sets and achieve improvements in the performance of $2.4\%$ and $2.7\%$ over standard single-supervision model training. Finally, we confirm the effectiveness of our approach on skin lesion classification problem.

Motivation & Objective

• To improve CNN generalization by incorporating domain-specific privileged information such as segmentation masks during training.
• To address the limitation of standard supervised learning, where models may learn spurious correlations due to lack of control over feature selection.
• To develop a method that guides model attention toward relevant image regions without altering the inference architecture.
• To verify the effectiveness of the approach on diverse benchmarks, including medical skin lesion classification.

Proposed method

• Introduces a parameter-free architectural component that uses VisualBackProp to back-propagate a desired visualization mask during training.
• Applies the visualization mask in reverse: instead of identifying salient features post-hoc, it enforces attention on specific regions during learning.
• Uses only linear operations—averaging, deconvolution, and point-wise multiplication—making it compatible with any CNN architecture.
• Applies the privileged mask at full resolution across all convolutional layers, suppressing irrelevant features like background or artifacts.
• Maintains the same network architecture at test time, ensuring no inference overhead.
• Employs a double-supervision framework: standard label supervision and privileged mask supervision during training.

Experimental results

Research questions

• RQ1 Can incorporating segmentation masks as privileged information improve CNN performance in image classification tasks?
• RQ2 How does VisualBackProp’s reverse visualization mechanism affect feature learning and model generalization?
• RQ3 Does the method outperform standard data augmentation and input masking techniques in terms of accuracy and robustness?
• RQ4 Can the method be effectively applied to medical image classification, where background artifacts (e.g., rulers, hairs) often mislead models?

Key findings

• The proposed method achieves a 2.4% absolute improvement in top-1 accuracy over standard single-supervision training on the ImageNet dataset.
• A 2.7% improvement in mean mAP is observed on the PASCAL VOC dataset compared to baseline training.
• On skin lesion classification, the Full Focus variant achieves an ROC-AUC of 0.815 for Melanoma, outperforming the regular training baseline (0.802).
• The Half Focus variant produces sharper visualization masks and achieves the highest AUC for Seborrheic Keratosis (0.941), surpassing the baseline (0.924).
• Exemplary visualizations confirm that the method effectively suppresses attention on background artifacts like hairs and rulers, focusing instead on the lesion region.
• The method incurs fewer computational operations than a single forward pass, making it highly efficient and deployable in real-world systems.