[Paper Review] On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models

Sven Gowal, Krishnamurthy Dvijotham+7 more|arXiv (Cornell University)|Oct 30, 2018

Adversarial Robustness in Machine Learning35 references300 citations

TL;DR

The paper shows that interval bound propagation (IBP) can be used to train large verifiably robust neural networks efficiently, achieving state-of-the-art verified accuracy on MNIST, CIFAR-10, SVHN, and non-vacuous verification on downscaled ImageNet.

ABSTRACT

Recent work has shown that it is possible to train deep neural networks that are provably robust to norm-bounded adversarial perturbations. Most of these methods are based on minimizing an upper bound on the worst-case loss over all possible adversarial perturbations. While these techniques show promise, they often result in difficult optimization procedures that remain hard to scale to larger networks. Through a comprehensive analysis, we show how a simple bounding technique, interval bound propagation (IBP), can be exploited to train large provably robust neural networks that beat the state-of-the-art in verified accuracy. While the upper bound computed by IBP can be quite weak for general networks, we demonstrate that an appropriate loss and clever hyper-parameter schedule allow the network to adapt such that the IBP bound is tight. This results in a fast and stable learning algorithm that outperforms more sophisticated methods and achieves state-of-the-art results on MNIST, CIFAR-10 and SVHN. It also allows us to train the largest model to be verified beyond vacuous bounds on a downscaled version of ImageNet.

Motivation & Objective

• Demonstrate that a simple interval bound propagation bound can be used to train verifiably robust classifiers at scale.
• Show that IBP can outperform more complex verification-based methods in verified accuracy while remaining computationally efficient.
• Provide a curriculum-based training strategy that stabilizes optimization and improves generalization for verifiably robust models.
• Evaluate IBP-based training against leading methods on MNIST, CIFAR-10, SVHN, and downscaled ImageNet to establish a new baseline.

Proposed method

• Propagate axis-aligned interval bounds through network layers to bound adversarial logits with respect to an l_infty perturbation bound.
• Formulate a training loss that combines nominal predictive loss with a specification loss derived from worst-case logits bound (L = kappa * L_fit + (1 - kappa) * L_spec).
• Elide the last linear layer to obtain tighter worst-case logits bounds (improves estimate of the worst-case logits).
• Use two forward passes through the network to compute IBP bounds, enabling fast, scalable bound propagation.
• Schedule curriculum: gradually increase epsilon during training and adjust kappa to balance fitting and verification objectives.
• Verify robustness with an exact MIP/LP cascade when feasible, and compare IBP bounds to these verifications.

Experimental results

Research questions

• RQ1 Can interval bound propagation provide a scalable and effective framework for training verifiably robust models?
• RQ2 How does IBP-trained robustness compare to state-of-the-art methods (e.g., Madry et al., Wong et al.) in terms of empirical and verified accuracy across datasets and epsilon values?
• RQ3 Are IBP bounds tight enough to serve as reliable proxies for full verification, and how does the tightness evolve during training?
• RQ4 Can IBP scale to larger networks and higher-resolution data (e.g., downscaled ImageNet) while maintaining non-vacuous verification?

Key findings

• IBP achieves state-of-the-art verified accuracy on MNIST, CIFAR-10, and SVHN across several perturbation radii (e.g., Mnist: 2.23% verified error at ε=0.1, 8.05% at ε=0.3; CIFAR-10: 67.96% verified error at ε=8/255).
• IBP scales to larger architectures and even to downscaled ImageNet (64×64) with a non-vacuous verified error of 93.87% at ε=1/255 for WideResNet-10-10.
• IBP bounds are competitive with, and often close to, full MIP/LP-based verification bounds, indicating the bound is a good proxy for verifiable robustness.
• IBP training is significantly faster than some alternatives (e.g., small-model training times ~3.5 s/epoch on a Titan Xp compared to minutes for some baselines).
• A carefully designed curriculum (epsilon and kappa) enables the model to adapt to IBP bounds and improve both nominal and verified performance.
• On ImageNet downscaled data, IBP provides verifiable robustness where no prior work demonstrated non-vacuous bounds at ε=1/255.