[Paper Review] Adversarially Robust Neural Architectures

Minjing Dong, Yanxi Li+2 more|arXiv (Cornell University)|Sep 2, 2020

Adversarial Robustness in Machine Learning42 references38 citations

TL;DR

This paper studies how neural architecture design affects adversarial robustness and introduces a learning-based architecture search with Lipschitz-based confidence constraints to find robust architectures.

ABSTRACT

Deep Neural Networks (DNNs) are vulnerable to adversarial attacks. Existing methods are devoted to developing various robust training strategies or regularizations to update the weights of the neural network. But beyond the weights, the overall structure and information flow in the network are explicitly determined by the neural architecture, which remains unexplored. This paper thus aims to improve the adversarial robustness of the network from the architecture perspective. We explore the relationship among adversarial robustness, Lipschitz constant, and architecture parameters and show that an appropriate constraint on architecture parameters could reduce the Lipschitz constant to further improve the robustness. The importance of architecture parameters could vary from operation to operation or connection to connection. We approximate the Lipschitz constant of the entire network through a univariate log-normal distribution, whose mean and variance are related to architecture parameters. The confidence can be fulfilled through formulating a constraint on the distribution parameters based on the cumulative function. Compared with adversarially trained neural architectures searched by various NAS algorithms as well as efficient human-designed models, our algorithm empirically achieves the best performance among all the models under various attacks on different datasets.

Motivation & Objective

• Investigate the relationship between adversarial robustness and network architecture under a NAS framework.
• Derive how architecture parameters influence the Lipschitz constant of the network.
• Propose a Lipschitz-constrained architecture search that samples architecture parameters from distributions.
• Introduce confidence learning to quantify and utilize uncertainty in architecture parameters during search.

Proposed method

• Approximate the network Lipschitz constant as a function of architecture parameters and derive a bound on the overall Lipschitz constant.
• Represent architecture via differentiable cells with operation and input weights, and unfold Lipschitz propagation across cells.
• Sample architecture parameters from multivariate log-normal distributions to enable confidence learning.
• Formulate a Lipschitz constraint as a cumulative distribution constraint using the log-normal approximation and a confidence hyperparameter η.
• Solve the constrained optimization via ADMM to jointly optimize weights and architecture distribution parameters.
• Provide an efficient NAS framework (RACL) that compares against adversarially trained NAS and hand-crafted models under various attacks.

Experimental results

Research questions

• RQ1 How does neural architecture impact adversarial robustness through the Lipschitz constant?
• RQ2 Can sampling architecture parameters from distributions and enforcing Lipschitz-based constraints improve robustness without excessive adversarial training?
• RQ3 What is the effect of confidence learning on exploring robust architectural paths within the NAS search space?
• RQ4 How can a differentiable NAS framework be augmented to incorporate Lipschitz constraints and confidence in architecture parameters?

Key findings

• The proposed framework links robustness to architecture via Lipschitz constant propagation across NAS cells.
• Using log-normal distributions to sample architecture parameters allows a differentiable, confidence-aware search.
• A convex Lipschitz constraint is derived and enforced through a probability-based (CDF) formulation with a confidence level η.
• RACL achieves superior robustness compared to other NAS approaches and state-of-the-art models across multiple datasets and attacks.
• Confidence learning helps the search prefer architectural paths with higher reliability while exploring lower-confidence options.