[Paper Review] Input Hessian Regularization of Neural Networks

Waleed Mustafa, Robert A. Vandermeulen+1 more|arXiv (Cornell University)|Sep 14, 2020

Adversarial Robustness in Machine Learning29 references4 citations

TL;DR

This paper proposes Cross-Hölder Regularization, a second-order Hessian operator-norm regularizer that improves deep neural network robustness against adversarial attacks by penalizing large input Hessian eigenvalues. The method efficiently computes gradients of the Hessian operator norm via a novel algorithm and demonstrates, on MNIST and FMNIST, that it outperforms both gradient-based regularization and adversarial training in robustness under strong attacks, especially at higher robustness levels beyond 4.0.

ABSTRACT

Regularizing the input gradient has shown to be effective in promoting the robustness of neural networks. The regularization of the input's Hessian is therefore a natural next step. A key challenge here is the computational complexity. Computing the Hessian of inputs is computationally infeasible. In this paper we propose an efficient algorithm to train deep neural networks with Hessian operator-norm regularization. We analyze the approach theoretically and prove that the Hessian operator norm relates to the ability of a neural network to withstand an adversarial attack. We give a preliminary experimental evaluation on the MNIST and FMNIST datasets, which demonstrates that the new regularizer can, indeed, be feasible and, furthermore, that it increases the robustness of neural networks over input gradient regularization.

Motivation & Objective

• To improve the robustness of deep neural networks against adversarial attacks by leveraging second-order derivative information.
• To address the computational infeasibility of directly computing input Hessian gradients by developing an efficient algorithm for Hessian operator-norm regularization.
• To theoretically link adversarial robustness to the Hessian operator norm, extending prior work based on gradient norms.
• To empirically validate that Hessian regularization enhances robustness beyond input gradient regularization, particularly under strong adversarial attacks.
• To provide a feasible and effective alternative to adversarial training and gradient-based defenses using second-order curvature information.

Proposed method

• Proposes Cross-Hölder Regularization, a second-order regularizer that penalizes the operator norm of the input Hessian matrix to control model curvature.
• Develops an efficient algorithm to compute the gradient of the Hessian operator norm, overcoming the computational infeasibility of naive approaches.
• Uses the Hessian operator norm as a proxy for local curvature, with large values indicating high sensitivity to input perturbations.
• Applies the regularizer during training by adding a penalty term proportional to the Hessian operator norm to the loss function.
• Replaces ReLU and max-pooling layers with differentiable alternatives (SWISH and strided convolutions) to ensure twice differentiability for Hessian computation.
• Employs a grid search on a held-out validation set to tune hyperparameters (λ₁, λ₂) for optimal clean accuracy and adversarial robustness.

Experimental results

Research questions

• RQ1 Can second-order derivatives, specifically the Hessian operator norm, be used to improve adversarial robustness in deep neural networks?
• RQ2 Is it computationally feasible to train deep networks with Hessian operator-norm regularization?
• RQ3 Does Hessian regularization outperform gradient-based regularization (e.g., Cross-Lipschitz) in robustness against strong adversarial attacks?
• RQ4 How does Hessian regularization compare to adversarial training in robustness under diverse attack settings?
• RQ5 What is the theoretical relationship between the Hessian operator norm and the robustness of a classifier under adversarial perturbations?

Key findings

• On the MNIST dataset, Cross-Hölder regularization outperforms both Cross-Lipschitz (gradient-based) regularization and adversarial training under the worst-case PGD attack with the Carlini-Wagner objective, especially at robustness levels above 4.0.
• At robustness level 4.0, Cross-Hölder achieved higher adversarial accuracy than both Cross-Lipschitz and adversarial training under the CW loss attack, demonstrating superior robustness to strong attacks.
• The method maintained over 99% clean accuracy on MNIST and 88% on FMNIST, indicating minimal degradation in standard performance.
• Empirical results show that Hessian regularization provides better robustness than gradient regularization, particularly when the attack is stronger and more sophisticated.
• The proposed algorithm enables efficient computation of the Hessian operator norm gradient, making second-order regularization feasible for deep networks.
• Theoretical analysis confirms that the Hessian operator norm bounds the robustness of a classifier, extending prior gradient-based bounds to include curvature effects.