[Paper Review] Vulnerability Analysis of Chest X-Ray Image Classification Against Adversarial Attacks

Saeid Asgari Taghanaki, Arkadeep Das+1 more|arXiv (Cornell University)|Jul 9, 2018

Adversarial Robustness in Machine Learning23 references4 citations

TL;DR

This paper evaluates the vulnerability of two state-of-the-art deep learning models—Inception-ResNet-v2 and Nasnet-Large—on chest X-ray image classification against ten adversarial attacks across white-box and black-box settings. It finds that gradient-based attacks are most effective, while replacing max-pooling with average-pooling significantly improves resilience, especially against decision- and score-based attacks, with Nasnet-Large using average pooling showing 18% higher robustness.

ABSTRACT

Recently, there have been several successful deep learning approaches for automatically classifying chest X-ray images into different disease categories. However, there is not yet a comprehensive vulnerability analysis of these models against the so-called adversarial perturbations/attacks, which makes deep models more trustful in clinical practices. In this paper, we extensively analyzed the performance of two state-of-the-art classification deep networks on chest X-ray images. These two networks were attacked by three different categories (ten methods in total) of adversarial methods (both white- and black-box), namely gradient-based, score-based, and decision-based attacks. Furthermore, we modified the pooling operations in the two classification networks to measure their sensitivities against different attacks, on the specific task of chest X-ray classification.

Motivation & Objective

• To assess the vulnerability of deep learning models in chest X-ray classification to adversarial perturbations.
• To evaluate the effectiveness of diverse adversarial attack types—gradient-based, score-based, and decision-based—on medical imaging models.
• To investigate whether modifying pooling operations (max vs. average) enhances model robustness against adversarial attacks.
• To compare model performance under white-box and black-box attack scenarios on the ChestX-ray14 dataset.
• To examine the perceptibility of adversarial examples to human observers and their impact on clinical trust.

Proposed method

• Trained and evaluated two state-of-the-art models: Inception-ResNet-v2 and Nasnet-Large on the ChestX-ray14 dataset.
• Applied ten adversarial attack methods across three categories: five gradient-based (FGSM, PGD, DeepFool, Linf-BIM, L-BFGS), two score-based (Local Search, Single Pixel), and three decision-based (Gaussian Blur, Contrast Reduction, Additive Gaussian Noise).
• Conducted both white-box attacks (same model for crafting and classifying) and black-box attacks (different models for crafting and classifying).
• Modified the networks by replacing max-pooling with average-pooling to assess sensitivity to adversarial perturbations.
• Measured model performance using accuracy (Acc.) and area under the ROC curve (AUROC) on clean and perturbed images.
• Visualized adversarial example perceptibility using human evaluation categories: Obvious, Mixed, and Hard.

Experimental results

Research questions

• RQ1 How effective are gradient-based, score-based, and decision-based adversarial attacks on chest X-ray classification models?
• RQ2 Does replacing max-pooling with average-pooling improve model robustness against adversarial attacks?
• RQ3 How does model performance degrade under white-box versus black-box attack settings?
• RQ4 To what extent are adversarial examples perceptible to human observers, and how does this correlate with attack success?
• RQ5 Which attack type is most effective in simultaneously fooling both the model and human perception?

Key findings

• Gradient-based attacks (e.g., PGD, L-BFGS) achieved near-total success in white-box settings, reducing accuracy to 43–46% across both models.
• Average-pooling improved robustness: Nasnet-Large with average-pooling showed 18% higher resilience compared to its max-pooling counterpart.
• Score-based attacks like Local Search (S1) failed completely against Nasnet-Large with average-pooling, indicating high sensitivity to pooling type.
• Decision-based attacks (e.g., Additive Gaussian Noise, Contrast Reduction) were detectable by humans in 23–30% of cases, with perceptibility levels ranging from 'Mixed' to 'Obvious'.
• Gradient-based attacks were the most effective in fooling both models and human observers, with 27% of test samples failing under max-pooling and 23% under average-pooling.
• In black-box settings, gradient-based attacks remained the most effective, while average-pooling models showed higher robustness, particularly against score- and decision-based attacks.