[Paper Review] Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability

Soyoun Won, Sung‐Ho Bae+1 more|arXiv (Cornell University)|Mar 26, 2023

Explainable Artificial Intelligence (XAI)Computer Science3 citations

TL;DR

This paper investigates the impact of mixed sample data augmentation on deep neural network interpretability, proposing three evaluation criteria: human alignment, model faithfulness, and disentangled concept detection. It finds that CutMix and SaliencyMix degrade interpretability despite boosting performance, suggesting caution in mission-critical applications like medical AI.

ABSTRACT

Mixed sample data augmentation strategies are actively used when training deep neural networks (DNNs). Recent studies suggest that they are effective at various tasks. However, the impact of mixed sample data augmentation on model interpretability has not been widely studied. In this paper, we explore the relationship between model interpretability and mixed sample data augmentation, specifically in terms of feature attribution maps. To this end, we introduce a new metric that allows a comparison of model interpretability while minimizing the impact of occlusion robustness of the model. Experimental results show that several mixed sample data augmentation decreases the interpretability of the model and label mixing during data augmentation plays a significant role in this effect. This new finding suggests it is important to carefully adopt the mixed sample data augmentation method, particularly in applications where attribution map-based interpretability is important.

Motivation & Objective

• To investigate the unexplored effect of mixed sample data augmentation on deep neural network interpretability.
• To address the lack of metrics for comparing interpretability across different models trained with varying augmentation strategies.
• To evaluate interpretability through human alignment, faithfulness, and detectable disentangled concepts in model internals.
• To provide a quantitative framework for inter-model interpretability comparison, supplementing existing method-centric metrics.
• To guide practitioners in selecting data augmentation strategies that balance performance and interpretability, especially in safety-critical domains.

Proposed method

• Proposed new inter-model interpretability evaluation metrics: Inter-model deletion and Inter-model insertion, enabling comparison across different models.
• Evaluated model interpretability using three criteria: (1) alignment with human-annotated bounding boxes in attribution maps, (2) faithfulness via deletion/insertion scores, and (3) number of detectable disentangled object concepts.
• Trained ResNet-50 on ImageNet using Cutout, Mixup, CutMix, and SaliencyMix for controlled comparison.
• Used concept activation theory (CAM) and object detector analysis to quantify disentanglement of learned features.
• Applied saliency-guided patch selection in SaliencyMix to prioritize semantically relevant regions during mixing.
• Conducted ablation studies comparing attribution map sparsity, localization accuracy, and concept detector counts across augmentation methods.

Experimental results

Research questions

• RQ1 How does mixed sample data augmentation affect the alignment of model attribution maps with human-annotated ground truth?
• RQ2 To what extent does data augmentation influence the faithfulness of feature attribution methods in post-hoc explanations?
• RQ3 Does mixed sample augmentation reduce the number of detectable disentangled concepts in model activations?
• RQ4 Are there trade-offs between model performance and interpretability when using CutMix and SaliencyMix?
• RQ5 Can inter-model interpretability be quantitatively measured and compared using deletion/insertion-based metrics across different augmentation strategies?

Key findings

• Models trained with CutMix and SaliencyMix exhibit significantly lower alignment with human-annotated bounding boxes in attribution maps compared to other augmentation methods.
• Inter-model deletion and insertion scores show that attribution maps from CutMix- and SaliencyMix-trained models are less faithful to the model's actual decision process.
• Despite achieving state-of-the-art performance (SaliencyMix: 21.26% top-1 error on ImageNet), models trained with SaliencyMix show the fewest number of detectable disentangled object concepts.
• CutMix and SaliencyMix reduce the number of unique object detectors by up to 30% compared to baseline models, indicating poorer feature disentanglement.
• The performance gain from mixed sample augmentation does not translate to improved interpretability, as evidenced across all three evaluation criteria.
• The data distribution shift induced by mixing samples appears to hinder the emergence of neurons that detect distinct, human-recognizable concepts.