[Paper Review] Towards Verifying the Geometric Robustness of Large-scale Neural Networks

Fu Wang, Peipei Xu+2 more|arXiv (Cornell University)|Jan 29, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes GeoRobust, a black-box robustness analyzer that uses Lipschitzian optimization to provably verify geometric robustness of large-scale neural networks against adversarial combinations of transformations like rotation and scaling. It efficiently locates worst-case transformations in seconds on ImageNet models, revealing that deeper and wider networks exhibit higher geometric robustness, with performance positively correlated to parameter count.

ABSTRACT

Deep neural networks (DNNs) are known to be vulnerable to adversarial geometric transformation. This paper aims to verify the robustness of large-scale DNNs against the combination of multiple geometric transformations with a provable guarantee. Given a set of transformations (e.g., rotation, scaling, etc.), we develop GeoRobust, a black-box robustness analyser built upon a novel global optimisation strategy, for locating the worst-case combination of transformations that affect and even alter a network's output. GeoRobust can provide provable guarantees on finding the worst-case combination based on recent advances in Lipschitzian theory. Due to its black-box nature, GeoRobust can be deployed on large-scale DNNs regardless of their architectures, activation functions, and the number of neurons. In practice, GeoRobust can locate the worst-case geometric transformation with high precision for the ResNet50 model on ImageNet in a few seconds on average. We examined 18 ImageNet classifiers, including the ResNet family and vision transformers, and found a positive correlation between the geometric robustness of the networks and the parameter numbers. We also observe that increasing the depth of DNN is more beneficial than increasing its width in terms of improving its geometric robustness. Our tool GeoRobust is available at https://github.com/TrustAI/GeoRobust.

Motivation & Objective

• To address the lack of provable, scalable methods for verifying robustness against combined geometric transformations in large-scale DNNs.
• To develop a black-box method that does not require architectural or gradient access to the target model.
• To provide theoretical convergence guarantees using Lipschitz continuity and optimize for GPU-parallel efficiency.
• To empirically benchmark geometric robustness across modern ImageNet classifiers, including ResNets and vision transformers.
• To investigate the impact of model depth and width on geometric robustness under adversarial transformations.

Proposed method

• GeoRobust employs a global optimization strategy based on Lipschitzian theory to provably locate the worst-case combination of geometric transformations.
• It proves that spatial transformation networks (STNs) are Lipschitz continuous, enabling theoretical convergence guarantees when stacked with Lipschitz-continuous DNNs.
• The framework uses a space-division strategy with adaptive sampling to efficiently explore the continuous parameter space of transformations.
• It introduces a GPU-aware sampling strategy to accelerate function queries and enables parallel evaluation across multiple transformation candidates.
• An anytime lower bound estimation method is used to provide verifiable confidence intervals at any stopping time, enhancing practical usability.
• The method operates as a black-box, requiring only input-output queries, making it applicable to any DNN regardless of architecture or activation function.

Experimental results

Research questions

• RQ1 Can a black-box, provably convergent method efficiently locate the worst-case combination of geometric transformations that fool large-scale DNNs?
• RQ2 How does the Lipschitz continuity of STNs enable theoretical guarantees in geometric robustness verification?
• RQ3 What is the relationship between model size (depth and width) and geometric robustness across modern ImageNet classifiers?
• RQ4 How does GeoRobust’s performance scale with the number of transformation parameters and model complexity?
• RQ5 To what extent do adversarial geometric attacks exploit architectural weaknesses not mitigated by standard adversarial training?

Key findings

• GeoRobust locates the worst-case geometric transformation for a ResNet50 model on ImageNet in an average of 6.9 to 16.1 seconds, depending on configuration.
• The geometric robustness of DNNs shows a positive correlation with the number of parameters across 18 ImageNet classifiers, including ResNets and vision transformers.
• Increasing network depth improves geometric robustness more effectively than increasing width, suggesting depth is a more efficient design choice for robustness.
• The method achieves sub-linear runtime growth with query count due to efficient GPU parallelization, enabling fast convergence.
• The estimated lower bound on the worst-case loss continuously improves during execution, demonstrating the effectiveness of the anytime estimation strategy.
• Adversarial training does not significantly improve geometric robustness, indicating that current defense methods may not be effective against geometric attacks.