[论文解读] Fast Certified Robust Training with Short Warmup

Zhouxing Shi, Yihan Wang+3 more|arXiv (Cornell University)|Mar 31, 2021

Adversarial Robustness in Machine Learning参考文献 54被引用 6

一句话总结

该论文提出了一种基于区间边界传播（IBP）的快速认证鲁棒训练方法，采用简短的预热训练策略。该方法引入了IBP专用的权重初始化、每一层的完整批量归一化（Batch Normalization）以及正则化技术，以稳定认证边界并平衡ReLU激活状态，仅用160和80个训练周期便在CIFAR-10（65.03%）和TinyImageNet（82.36%）上实现了当前最优的验证错误率，显著降低了训练成本。

ABSTRACT

Recently, bound propagation based certified robust training methods have been proposed for training neural networks with certifiable robustness guarantees. Despite that state-of-the-art (SOTA) methods including interval bound propagation (IBP) and CROWN-IBP have per-batch training complexity similar to standard neural network training, they usually use a long warmup schedule with hundreds or thousands epochs to reach SOTA performance and are thus still costly. In this paper, we identify two important issues in existing methods, namely exploded bounds at initialization, and the imbalance in ReLU activation states and improve IBP training. These two issues make certified training difficult and unstable, and thereby long warmup schedules were needed in prior works. To mitigate these issues and conduct faster certified training with shorter warmup, we propose three improvements based on IBP training: 1) We derive a new weight initialization method for IBP training; 2) We propose to fully add Batch Normalization (BN) to each layer in the model, since we find BN can reduce the imbalance in ReLU activation states; 3) We also design regularization to explicitly tighten certified bounds and balance ReLU activation states during wamrup. We are able to obtain 65.03% verified error on CIFAR-10 ($ε=\frac{8}{255}$) and 82.36% verified error on TinyImageNet ($ε=\frac{1}{255}$) using very short training schedules (160 and 80 total epochs, respectively), outperforming literature SOTA trained with hundreds or thousands epochs under the same network architecture. The code is available at https://github.com/shizhouxing/Fast-Certified-Robust-Training.

研究动机与目标

• 解决由于初始化时认证边界爆炸和ReLU激活状态失衡导致的认证鲁棒训练不稳定与高成本问题。
• 减少IBP-based认证训练中对长预热周期（数百至数千个周期）的依赖。
• 在保持或超越当前最先进认证鲁棒性性能的同时，提升训练效率。
• 识别并解决IBP训练中的关键不稳定性，包括权重初始化不佳和ReLU激活状态失衡。

提出的方法

• 提出一种新的IBP专用权重初始化方法，以在初始化阶段稳定认证边界的紧致性。
• 在每一层引入完整的批量归一化，以平衡ReLU激活状态并稳定梯度流动。
• 设计正则化项，显式收紧认证边界，并在预热阶段维持ReLU激活状态的平衡。
• 将基于IBP的认证损失与简短的预热周期结合，逐步将扰动半径从0增加到目标值。
• 采用改进的训练目标，以稳定方差并防止早期训练中边界爆炸。
• 使用统一框架整合IBP初始化、批量归一化与正则化，实现稳定且快速的认证鲁棒训练。

实验结果

研究问题

• RQ1 为何现有IBP-based认证训练方法需要长预热周期？
• RQ2 不良权重初始化在IBP训练中如何导致认证边界传播不稳定？
• RQ3 批量归一化在认证训练中能在多大程度上缓解ReLU激活状态失衡？
• RQ4 正则化是否能有效收紧认证边界，并在短预热阶段提升训练稳定性？
• RQ5 在不损害性能的前提下，实现最先进认证鲁棒性的最小训练周期是多少？

主要发现

• 所提方法在CIFAR-10上仅用160个总训练周期（ϵ = 8/255）即实现65.03%的验证错误率。
• 在TinyImageNet上（ϵ = 1/255）仅用80个周期即实现82.36%的验证错误率，优于先前的SOTA方法。
• 与先前使用900–2000个周期的同类架构相比，该方法显著降低了总训练成本。
• 完整批量归一化有效减少了ReLU激活状态失衡，提升了训练稳定性和性能。
• IBP专用权重初始化可防止初始化阶段的边界爆炸，实现更快收敛。
• IBP初始化、完整批量归一化与正则化的结合，使最小预热下的稳定训练成为可能。