[Paper Review] 3D Common Corruptions and Data Augmentation

Oğuzhan Fatih Kar, Teresa Yeo+2 more|arXiv (Cornell University)|Mar 2, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces 3D Common Corruptions (3DCC), a set of geometry-aware image corruptions that simulate real-world distortions like motion blur with depth parallax, defocus blur with depth of field, and occlusions based on 3D scene structure. The method improves model robustness when used as data augmentation, reducing error by up to 20% on 3DCC and AE benchmarks compared to 2D-only augmentations.

ABSTRACT

We introduce a set of image transformations that can be used as corruptions to evaluate the robustness of models as well as data augmentation mechanisms for training neural networks. The primary distinction of the proposed transformations is that, unlike existing approaches such as Common Corruptions, the geometry of the scene is incorporated in the transformations -- thus leading to corruptions that are more likely to occur in the real world. We also introduce a set of semantic corruptions (e.g. natural object occlusions). We show these transformations are `efficient' (can be computed on-the-fly), `extendable' (can be applied on most image datasets), expose vulnerability of existing models, and can effectively make models more robust when employed as `3D data augmentation' mechanisms. The evaluations on several tasks and datasets suggest incorporating 3D information into benchmarking and training opens up a promising direction for robustness research.

Motivation & Objective

• To develop a benchmark of realistic, geometry-aware image corruptions that reflect real-world distribution shifts in 3D scenes.
• To address the limitations of 2D-only corruptions like Common Corruptions, which ignore scene depth and produce implausible distortions.
• To design efficient, on-the-fly 3D data augmentation techniques that incorporate 3D scene geometry to improve model robustness.
• To demonstrate that 3DCC exposes vulnerabilities in existing models and improves robustness when used during training.
• To provide a scalable, extendable framework applicable to standard vision datasets without requiring 3D annotations.

Proposed method

• Generates corruptions using 3D scene geometry from monocular depth estimates, enabling realistic effects like motion parallax and depth-of-field blur.
• Applies scene-aware transformations such as depth-dependent fog, lighting with shadows, and viewpoint-based occlusions using 3D scene reconstruction.
• Introduces 20 distinct 3D corruptions, including defocus blur, motion blur, lighting changes, and semantic occlusions, all computed on-the-fly.
• Employs differentiable rendering and 3D-aware warping to apply corruptions while preserving spatial consistency with scene geometry.
• Uses a dual training strategy: evaluate model robustness on 3DCC and train with 3D augmentations to learn invariance to geometric distortions.
• Supports extension to standard datasets (e.g., ImageNet, Taskonomy) by leveraging estimated depth maps, enabling broad applicability without 3D labels.

Experimental results

Research questions

• RQ1 How do 3D-aware corruptions differ from traditional 2D corruptions in terms of realism and failure mode exposure?
• RQ2 To what extent do existing robustness methods fail under 3D Common Corruptions compared to 2D corruptions?
• RQ3 Can 3D-aware data augmentation significantly improve model robustness on real-world distribution shifts?
• RQ4 How do 3DCC-based augmentations compare to 2D-only augmentations in improving performance on clean and corrupted data?
• RQ5 Can 3DCC serve as a reliable benchmark for evaluating generalization under geometrically plausible corruptions?

Key findings

• The 3DCC benchmark exposes significant robustness failures in models trained with 2D data augmentations, with performance drops of up to 20% on 3DCC compared to 2DCC.
• Models trained with 3D data augmentation (Ours) achieve 5.32 ℓ₁ error on 2DCC and 5.42 ℓ₁ error on 3DCC, outperforming O+DPT+2DCC (5.78 and 5.94) and O+DPT (6.43 and 6.13).
• On the AE benchmark (synthetic real-world corruptions), the proposed model achieves 4.94 ℓ₁ error, a 23.9% improvement over O+DPT+2DCC (6.50).
• On the OASIS dataset, the proposed model achieves 23.89 angular error, a 4.9% improvement over the next best baseline (24.65), demonstrating robustness on real-world in-the-wild data.
• The Ours+X-TC model further reduces error to 5.29 on 2DCC and 5.35 on 3DCC, showing that combining 3D augmentation with cross-task consistency improves generalization.
• Qualitative results show that predictions from models trained with 3D augmentations are significantly more robust across diverse real-world scenarios, including DSLR images and YouTube videos.