[论文解读] Towards Fast Computation of Certified Robustness for ReLU Networks

Tsui-Wei Weng, Huan Zhang+6 more|arXiv (Cornell University)|Apr 25, 2018

Adversarial Robustness in Machine Learning参考文献 49被引用 152

一句话总结

本文提出两种快速算法 Fast-Lin 和 Fast-Lip，用以计算 ReLU 网络对最小对抗性扭曲的有证下界，相较于以往的精确方法，提供更快且更紧的保证。

ABSTRACT

Verifying the robustness property of a general Rectified Linear Unit (ReLU) network is an NP-complete problem [Katz, Barrett, Dill, Julian and Kochenderfer CAV17]. Although finding the exact minimum adversarial distortion is hard, giving a certified lower bound of the minimum distortion is possible. Current available methods of computing such a bound are either time-consuming or delivering low quality bounds that are too loose to be useful. In this paper, we exploit the special structure of ReLU networks and provide two computationally efficient algorithms Fast-Lin and Fast-Lip that are able to certify non-trivial lower bounds of minimum distortions, by bounding the ReLU units with appropriate linear functions Fast-Lin, or by bounding the local Lipschitz constant Fast-Lip. Experiments show that (1) our proposed methods deliver bounds close to (the gap is 2-3X) exact minimum distortion found by Reluplex in small MNIST networks while our algorithms are more than 10,000 times faster; (2) our methods deliver similar quality of bounds (the gap is within 35% and usually around 10%; sometimes our bounds are even better) for larger networks compared to the methods based on solving linear programming problems but our algorithms are 33-14,000 times faster; (3) our method is capable of solving large MNIST and CIFAR networks up to 7 layers with more than 10,000 neurons within tens of seconds on a single CPU core. In addition, we show that, in fact, there is no polynomial time algorithm that can approximately find the minimum $\ell_1$ adversarial distortion of a ReLU network with a $0.99\ln n$ approximation ratio unless $\mathsf{NP}$=$\mathsf{P}$, where $n$ is the number of neurons in the network.

研究动机与目标

• 推动对 ReLU 网络的有证鲁棒性验证，并解决找到精确最小扭曲的 NP-hard 性质。
• 为实际网络规模提供快速的、有证的最小对抗性扭曲下界。
• 利用 ReLU 的结构来推导具有保证的可扩展界定技术。

提出的方法

• 通过对 ReLU 激活计算显式的线性上界和下界，以在有界扰动下获得有证的输出界限，进而开发 Fast-Lin。
• 通过对局部李氏常数进行上界来开发 Fast-Lip，以对扭曲提供下界的有证证明。
• 定义激活模式分区并构造矩阵，以在跨层之间表达精确的解析界限。
• 提供封闭式、分层界限，使得可在多项式时间内计算，避免 SMT/MILP 求解器。
• 对扰动半径进行二分搜索，以从输出界限获得最大的有证下界。

实验结果

研究问题

• RQ1 我们是否能在大规模下高效地为 ReLU 网络计算非平凡的、有证的最小对抗性扭曲下界？
• RQ2 相较于 Reluplex 和基于 LP 的方法，快速界定方法能达到多接近的确切最小扭曲？
• RQ3 在 ReLU 网络中获得 L1 鲁棒性的紧界逼近保证的理论难度是什么？
• RQ4 如何利用 ReLU 激活结构推导出可扩展且可证明的鲁棒性证书？

主要发现

• Fast-Lin 的界限与 Reluplex 找到的精确最小扭曲的差距在小型网络中为 2-3 倍，且速度超过 10,000 倍。
• Fast-Lin 对更大网络提供与基于 LP 的方法相当的界限质量，差距在 35% 之内，通常约 10%，同时速度提高 33–14,000 倍。
• Fast-Lip 通过对局部李氏常数的上界来提供有证下界，使在 CPU 上对最多 7 层、超过 10,000 个神经元的网络进行可扩展的认证在数十秒内完成。
• 这些方法使得对大型 MNIST 和 CIFAR 网络计算界限成为可能，且利用多项式时间计算，与 SMT/MILP 求解器不同。