[Paper Review] Bilateral Asymmetry Guided Counterfactual Generating Network for Mammogram Classification

Churan Wang, Jing Li+5 more|arXiv (Cornell University)|Sep 30, 2020

AI in cancer detectionComputer Science52 references20 citations

TL;DR

This paper proposes a Bilateral Asymmetry Guided Counterfactual Generating Network (CGN) for mammogram classification using only image-level labels. By leveraging the anatomical symmetry prior—where lesions are rare in corresponding areas of bilateral breasts—the method generates counterfactual features via a GAN-based framework that jointly optimizes lesion localization and classification through adversarial training and feedback from classification loss. The approach achieves state-of-the-art performance on INBreast and an in-house dataset, with a 91.0% feature-level accuracy and faster training than image-level generation.

ABSTRACT

Mammogram benign or malignant classification with only image-level labels is challenging due to the absence of lesion annotations. Motivated by the symmetric prior that the lesions on one side of breasts rarely appear in the corresponding areas on the other side, given a diseased image, we can explore a counterfactual problem that how would the features have behaved if there were no lesions in the image, so as to identify the lesion areas. We derive a new theoretical result for counterfactual generation based on the symmetric prior. By building a causal model that entails such a prior for bilateral images, we obtain two optimization goals for counterfactual generation, which can be accomplished via our newly proposed counterfactual generative network. Our proposed model is mainly composed of Generator Adversarial Network and a \emph{prediction feedback mechanism}, they are optimized jointly and prompt each other. Specifically, the former can further improve the classification performance by generating counterfactual features to calculate lesion areas. On the other hand, the latter helps counterfactual generation by the supervision of classification loss. The utility of our method and the effectiveness of each module in our model can be verified by state-of-the-art performance on INBreast and an in-house dataset and ablation studies.

Motivation & Objective

• To address mammogram benign/malignant classification using only image-level labels, avoiding costly lesion annotations.
• To exploit the anatomical symmetry prior—where lesions are unlikely to appear in corresponding bilateral breast regions—as a domain-specific inductive bias for lesion localization.
• To generate counterfactual features that simulate what the target image would look like without lesions, using a causal model grounded in symmetry.
• To jointly optimize counterfactual generation and classification via a feedback mechanism, improving both interpretability and performance.

Proposed method

• Proposes a structural causal model (SCM) with a hidden confounder to model bilateral symmetry, ensuring counterfactual features match reference features in lesion-free areas.
• Introduces a novel Counterfactual Generation Network (CGN) that operates in feature space to avoid pixel-level registration issues.
• Uses a generator-adversarial network with a feedback mechanism: classification loss guides counterfactual generation, while generated features improve classification.
• Employs a triplet loss (LF T) to enforce consistency between original, counterfactual, and reference features, with adversarial and reconstruction losses.
• Jointly optimizes multiple losses: LG (generator), LD (discriminator), LNE (normalization), LFC (feature consistency), and LCLS (classification).
• Applies horizontal flipping to reference images for alignment and uses max-pooling across 256 feature channels for visualization.

Experimental results

Research questions

• RQ1 Can the bilateral symmetry prior be formally leveraged to generate meaningful counterfactual features for lesion localization in mammograms?
• RQ2 How can counterfactual generation be formulated as a causal inference problem using a structural causal model (SCM) in medical imaging?
• RQ3 Can joint optimization of counterfactual generation and classification improve performance beyond supervised baselines with only image-level labels?
• RQ4 Does feature-level counterfactual generation outperform image-level generation in terms of accuracy and training speed?

Key findings

• The proposed CGN achieves a 91.0% AUC on the INBreast dataset for mass classification with AlexNet, outperforming all variants and baselines.
• On mixed-lesion classification with ResNet50, the method achieves 88.5% AUC on INBreast and 78.1% on the in-house dataset, demonstrating robustness across lesion types.
• The ablation study confirms that removing either the adversarial loss or the feedback triplet loss leads to performance degradation, validating the necessity of both components.
• Feature-level counterfactual generation achieves 91.0% accuracy with 6.6 seconds per epoch, significantly faster than image-level generation (23.5 seconds per epoch).
• Visualization shows that the generated counterfactual features effectively suppress lesion-related activations, while preserving contextual information in the residual features.
• The method outperforms SBF, GF, and SFF—existing bilateral comparison methods—by a clear margin, demonstrating the superiority of counterfactual learning over simple feature fusion.