[Paper Review] Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers

Katelyn Morrison, Benjamin Gilby+3 more|arXiv (Cornell University)|Jun 24, 2021

Adversarial Robustness in Machine Learning20 references9 citations

TL;DR

This paper investigates corruption robustness and shape bias in vision transformers, MLP-Mixers, and CNNs using pre-trained models on ImageNet-C and a texture-cue conflict dataset. It finds that vision transformers are inherently more robust to corruptions and exhibit stronger shape bias than CNNs and MLP-Mixers, with smaller models like DeiT-tiny outperforming ResNet-50 despite having five times fewer parameters.

ABSTRACT

Recently, vision transformers and MLP-based models have been developed in order to address some of the prevalent weaknesses in convolutional neural networks. Due to the novelty of transformers being used in this domain along with the self-attention mechanism, it remains unclear to what degree these architectures are robust to corruptions. Despite some works proposing that data augmentation remains essential for a model to be robust against corruptions, we propose to explore the impact that the architecture has on corruption robustness. We find that vision transformer architectures are inherently more robust to corruptions than the ResNet-50 and MLP-Mixers. We also find that vision transformers with 5 times fewer parameters than a ResNet-50 have more shape bias. Our code is available to reproduce.

Motivation & Objective

• To evaluate and compare corruption robustness across vision transformers, MLP-Mixers, and CNNs.
• To investigate the relationship between shape bias and robustness to image corruptions.
• To determine whether architectural inductive biases—particularly in self-attention mechanisms—contribute to robustness independently of data augmentation.
• To assess whether smaller vision transformer models can match or exceed larger CNNs in robustness and shape bias.

Proposed method

• Evaluation of 20 pre-trained models across three architectures: ResNet-50, vision transformers (ViT, DeiT, Swin, CaiT), and MLP-Mixers (Base, Large).
• Measurement of mean corruption error (mCE) on ImageNet-C to quantify corruption robustness.
• Assessment of shape bias using a texture-cue conflict dataset where shape and texture conflict.
• Correlation analysis between shape bias and mCE across models to identify architectural influences on robustness.
• Comparison of models with varying parameter counts to assess the role of model size in robustness and bias.
• Use of standard ImageNet top-1 accuracy and mCE to evaluate performance on clean and corrupted images.

Experimental results

Research questions

• RQ1 Do vision transformers exhibit greater robustness to common image corruptions than CNNs and MLP-Mixers?
• RQ2 Is there a correlation between shape bias and corruption robustness across different architectures?
• RQ3 Can smaller vision transformer models achieve better corruption robustness and shape bias than larger CNNs?
• RQ4 To what extent do architectural inductive biases—such as self-attention—contribute to robustness independently of data augmentation?
• RQ5 How do different vision transformer variants (e.g., Swin, DeiT, ViT) compare in terms of shape bias and mCE?

Key findings

• Vision transformers consistently outperform both CNNs and MLP-Mixers in corruption robustness, with the Swin-T_large model achieving the lowest mCE of 34.63%.
• The DeiT_tiny vision transformer, with only 5 million parameters, achieves a shape bias of 29.37%, surpassing ResNet-50 (26.17%) despite having five times fewer parameters.
• A strong inverse relationship exists between shape bias and mCE: models with higher shape bias are more robust to corruptions, supporting the hypothesis that shape bias enhances robustness.
• MLP-Mixers show moderate shape bias (36.90–38.64%) and perform similarly to CNNs on ImageNet-C, but are less robust than vision transformers.
• The Swin-T_large model achieves the highest top-1 accuracy (85.92%) and lowest mCE (34.63%), suggesting its hierarchical window-based self-attention improves global representation and robustness.
• No clear correlation was found between model size (number of parameters) and either shape bias or mCE, indicating that architecture design—not scale—drives robustness.