[논문 리뷰] On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models

Sven Gowal, Krishnamurthy Dvijotham외 7명|arXiv (Cornell University)|2018. 10. 30.

Adversarial Robustness in Machine Learning참고 문헌 35인용 수 300

한 줄 요약

논문은 간격 경계 전파(IBP)를 사용하여 대규모로 검증 가능 로버스트 신경망을 효율적으로 학습시킬 수 있음을 보이며, MNIST, CIFAR-10, SVHN에서 최첨단 검증 정확도와 downscaled ImageNet에서 비공허(non-vacuous) 검증을 달성한다.

ABSTRACT

Recent work has shown that it is possible to train deep neural networks that are provably robust to norm-bounded adversarial perturbations. Most of these methods are based on minimizing an upper bound on the worst-case loss over all possible adversarial perturbations. While these techniques show promise, they often result in difficult optimization procedures that remain hard to scale to larger networks. Through a comprehensive analysis, we show how a simple bounding technique, interval bound propagation (IBP), can be exploited to train large provably robust neural networks that beat the state-of-the-art in verified accuracy. While the upper bound computed by IBP can be quite weak for general networks, we demonstrate that an appropriate loss and clever hyper-parameter schedule allow the network to adapt such that the IBP bound is tight. This results in a fast and stable learning algorithm that outperforms more sophisticated methods and achieves state-of-the-art results on MNIST, CIFAR-10 and SVHN. It also allows us to train the largest model to be verified beyond vacuous bounds on a downscaled version of ImageNet.

연구 동기 및 목표

• 스케일에 맞춰 검증 가능 분류기를 학습시키기 위해 간단한 간격 경계 전파 bound를 사용할 수 있음을 입증한다.
• IBP가 검증 정확도 면에서 더 복잡한 검증 기반 방법을 능가하면서도 계산적으로 효율적임을 보여준다.
• 최적화를 안정시키고 검증 가능 모델의 일반화를 향상시키는 커리큘럼 기반 학습 전략을 제공한다.
• MNIST, CIFAR-10, SVHN, 및 downscaled ImageNet에 대해 IBP 기반 학습을 선도 방법들과 비교하여 새로운 기준점을 확립한다.]
• method:[
• 축에 정렬된 간격 경계를 네트워크 계층을 통과시키며 l_infty 섭동 bound에 대해 적대적 로짓을 바운드한다.
• 손실 함수는 명목 예측 손실과 worst-case 로짓 bound에서 파생된 명세 손실을 결합한 것을 사용한다 (L = kappa * L_fit + (1 - kappa) * L_spec).
• 마지막 선형 층을 생략하여 worst-case 로짓 bound를 더 촘촐하게 얻는다 (worst-case 로짓 추정 개선).
• IBP bound를 빠르고 확장 가능하게 계산하기 위해 네트워크를 두 번 순전파한다.
• 커리큘럼 스케줄링: 학습 도중 epsilon을 점진적으로 증가시키고 kappa를 조정해 적합성과 검증 목표의 균형을 맞춘다.
• 가능할 때 정확한 MIP/LP 캐스케이드를 사용해 로버스트함을 검증하고, IBP bound를 이 검증들과 비교한다.

제안 방법

• Propagate axis-aligned interval bounds through network layers to bound adversarial logits with respect to an l_infty perturbation bound.
• Formulate a training loss that combines nominal predictive loss with a specification loss derived from worst-case logits bound (L = kappa * L_fit + (1 - kappa) * L_spec).
• Elide the last linear layer to obtain tighter worst-case logits bounds (improves estimate of the worst-case logits).
• Use two forward passes through the network to compute IBP bounds, enabling fast, scalable bound propagation.
• Schedule curriculum: gradually increase epsilon during training and adjust kappa to balance fitting and verification objectives.
• Verify robustness with an exact MIP/LP cascade when feasible, and compare IBP bounds to these verifications.

실험 결과

연구 질문

• RQ1 Can interval bound propagation provide a scalable and effective framework for training verifiably robust models?
• RQ2 How does IBP-trained robustness compare to state-of-the-art methods (e.g., Madry et al., Wong et al.) in terms of empirical and verified accuracy across datasets and epsilon values?
• RQ3 Are IBP bounds tight enough to serve as reliable proxies for full verification, and how does the tightness evolve during training?
• RQ4 Can IBP scale to larger networks and higher-resolution data (e.g., downscaled ImageNet) while maintaining non-vacuous verification?

주요 결과

• IBP achieves state-of-the-art verified accuracy on MNIST, CIFAR-10, and SVHN across several perturbation radii (e.g., Mnist: 2.23% verified error at ε=0.1, 8.05% at ε=0.3; CIFAR-10: 67.96% verified error at ε=8/255).
• IBP scales to larger architectures and even to downscaled ImageNet (64×64) with a non-vacuous verified error of 93.87% at ε=1/255 for WideResNet-10-10.
• IBP bounds are competitive with, and often close to, full MIP/LP-based verification bounds, indicating the bound is a good proxy for verifiable robustness.
• IBP training is significantly faster than some alternatives (e.g., small-model training times ~3.5 s/epoch on a Titan Xp compared to minutes for some baselines).
• A carefully designed curriculum (epsilon and kappa) enables the model to adapt to IBP bounds and improve both nominal and verified performance.
• On ImageNet downscaled data, IBP provides verifiable robustness where no prior work demonstrated non-vacuous bounds at ε=1/255.