[Paper Review] Parseval Networks: Improving Robustness to Adversarial Examples

Moustapha Cissé, Piotr Bojanowski+3 more|arXiv (Cornell University)|Apr 28, 2017

Anomaly Detection Techniques and ApplicationsComputer Science179 citations

TL;DR

Parseval networks constrain layerwise Lipschitz constants by maintaining Parseval tight frames and convex aggregation to improve robustness to adversarial perturbations, while preserving or improving accuracy and training speed.

ABSTRACT

We introduce Parseval networks, a form of deep neural networks in which the Lipschitz constant of linear, convolutional and aggregation layers is constrained to be smaller than 1. Parseval networks are empirically and theoretically motivated by an analysis of the robustness of the predictions made by deep neural networks when their input is subject to an adversarial perturbation. The most important feature of Parseval networks is to maintain weight matrices of linear and convolutional layers to be (approximately) Parseval tight frames, which are extensions of orthogonal matrices to non-square matrices. We describe how these constraints can be maintained efficiently during SGD. We show that Parseval networks match the state-of-the-art in terms of accuracy on CIFAR-10/100 and Street View House Numbers (SVHN) while being more robust than their vanilla counterpart against adversarial examples. Incidentally, Parseval networks also tend to train faster and make a better usage of the full capacity of the networks.

Motivation & Objective

• Motivate robustness of deep networks against small input perturbations (adversarial examples).
• Introduce a layerwise regularization (Parseval regularization) to constrain Lipschitz constants.
• Develop an efficient training procedure compatible with SGD and common architectures (fully connected, conv, residual).
• Show that Parseval networks maintain competitive accuracy while improving adversarial robustness and training speed.

Proposed method

• Constrain the Lipschitz constant of each hidden layer to be <= 1 by maintaining approximately Parseval tight frames for weight matrices.
• For convolutional layers, constrain W to be a Parseval tight frame with output rescaled by (2k+1)^(-1/2).
• Replace standard aggregation (sum) with convex combinations of inputs to guarantee Lipschitz bounds via learned alphas on the simplex.
• Optimize weight matrices on the Stiefel-like manifold via a practical regularizer R_beta(W)= (beta/2)||W^T W - I||_2^2 with an efficient projection step.
• Apply a one-step retraction (and optional row sampling) to keep weights quasi-orthogonal during SGD updates.
• Use a simplex projection for aggregation coefficients to ensure Lambda_p <= 1 at each node.

Experimental results

Research questions

• RQ1 Can constraining layerwise Lipschitz constants via Parseval regularization improve robustness to adversarial perturbations without sacrificing accuracy?
• RQ2 How can we efficiently enforce Parseval constraints in SGD for fully connected, convolutional, and residual architectures?
• RQ3 What is the impact of combining Parseval regularization with adversarial training on robustness across standard image datasets?

Key findings

• Parseval training yields weight matrices with singular values tightly centered near 1, indicating near-orthogonality.
• Parseval networks deliver competitive clean accuracy on CIFAR-10/100 and SVHN compared to vanilla models.
• Parseval networks substantially improve robustness to adversarial examples, often outperforming vanilla models and matching or exceeding adversarial training in several settings.
• Combining Parseval regularization with adversarial training yields the most robust performance, especially at higher noise levels.
• Parseval networks tend to train faster than vanilla counterparts and make better use of network capacity.