[Paper Review] Measuring Robustness in Deep Learning Based Compressive Sensing

Mohammad Zalbagi Darestani, Akshay Chaudhari+1 more|arXiv (Cornell University)|Feb 11, 2021

Sparse and Compressive Sensing TechniquesEngineering53 references47 citations

TL;DR

The paper compares trained, un-trained, and traditional CS MRI reconstruction methods across adversarial perturbations, distribution shifts, and small-detail recovery to assess robustness beyond reconstruction quality.

ABSTRACT

Deep neural networks give state-of-the-art accuracy for reconstructing images from few and noisy measurements, a problem arising for example in accelerated magnetic resonance imaging (MRI). However, recent works have raised concerns that deep-learning-based image reconstruction methods are sensitive to perturbations and are less robust than traditional methods: Neural networks (i) may be sensitive to small, yet adversarially-selected perturbations, (ii) may perform poorly under distribution shifts, and (iii) may fail to recover small but important features in an image. In order to understand the sensitivity to such perturbations, in this work, we measure the robustness of different approaches for image reconstruction including trained and un-trained neural networks as well as traditional sparsity-based methods. We find, contrary to prior works, that both trained and un-trained methods are vulnerable to adversarial perturbations. Moreover, both trained and un-trained methods tuned for a particular dataset suffer very similarly from distribution shifts. Finally, we demonstrate that an image reconstruction method that achieves higher reconstruction quality, also performs better in terms of accurately recovering fine details. Our results indicate that the state-of-the-art deep-learning-based image reconstruction methods provide improved performance than traditional methods without compromising robustness.

Motivation & Objective

• Assess robustness of deep learning and traditional CS MRI reconstruction methods to adversarial perturbations.
• Examine sensitivity to distribution shifts across datasets and anatomies.
• Investigate ability to recover small but clinically relevant image details.
• Compare trained neural networks, un-trained networks, and classical sparsity-based methods.
• Provide guidance on robustness-performance trade-offs in MRI reconstruction.

Proposed method

• Study three method families: trained networks (U-net, VarNet), un-trained networks (Deep Decoder variants), and traditional ℓ1-wavelet sparsity with ESPIRiT coil maps.
• Generate adversarial perturbations tailored to each method and measure reconstruction loss under ℓ2-norm constrained perturbations.
• Evaluate distribution shifts via dataset shift (fastMRI knee to Stanford knee), anatomy shift (knee to brain and vice versa), and adversarially-filtered shifts (fastMRI-A).
• Quantify small-feature recovery by artificial 3x3 feature insertions and by real annotated pathologies in knee images.
• Analyze robustness of each method to distribution shifts and perturbations using SSIM, PSNR, and region-based MSE metrics.

Experimental results

Research questions

• RQ1 Do trained and un-trained MRI reconstruction methods exhibit similar vulnerabilities to small adversarial perturbations?
• RQ2 How do distribution shifts affect reconstruction quality across method families, and is out-of-distribution performance correlated with in-distribution performance?
• RQ3 Is there a trade-off between overall reconstruction quality and the ability to recover fine image details across methods?
• RQ4 How do small or clinically relevant features recover across different reconstruction approaches?
• RQ5 Do un-trained methods show comparable robustness to trained networks under dataset, anatomy, and adversarial shifts?

Key findings

• All methods, including trained networks, un-trained networks, and traditional ℓ1-based CS, are vulnerable to small adversarial perturbations.
• Adversarial perturbations tailored to one method can significantly impair that method, while having milder effects on others, indicating method-specific robustness weaknesses.
• Distribution shifts (dataset, anatomy, adversarially-filtered) degrade performance across both trained and un-trained methods with similar absolute drops.
• Out-of-distribution performance is linearly correlated with in-distribution performance, and ranking of methods remains similar under shifts.
• Higher in-distribution reconstruction quality correlates with better recovery of fine details and small features in the image.
• Un-trained methods are not inherently more robust to shifts, and the best-performing method in reconstruction quality also tends to perform best in small feature recovery.