[论文解读] Branch and Bound for Piecewise Linear Neural Network Verification

Rudy Bunel, Jingyue Lu+4 more|arXiv (Cornell University)|Sep 14, 2019

Adversarial Robustness in Machine Learning参考文献 27被引用 41

一句话总结

论文提出一种用于验证分段线性神经网络的分支限界（Branch-and-Bound）框架，统一了先前方法并提供新的分支与界定策略，尤其针对卷积网络架构。

ABSTRACT

The success of Deep Learning and its potential use in many safety-critical applications has motivated research on formal verification of Neural Network (NN) models. In this context, verification involves proving or disproving that an NN model satisfies certain input-output properties. Despite the reputation of learned NN models as black boxes, and the theoretical hardness of proving useful properties about them, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure and taking insights from formal methods such as Satisifiability Modulo Theory. However, these methods are still far from scaling to realistic neural networks. To facilitate progress on this crucial area, we exploit the Mixed Integer Linear Programming (MIP) formulation of verification to propose a family of algorithms based on Branch-and-Bound (BaB). We show that our family contains previous verification methods as special cases. With the help of the BaB framework, we make three key contributions. Firstly, we identify new methods that combine the strengths of multiple existing approaches, accomplishing significant performance improvements over previous state of the art. Secondly, we introduce an effective branching strategy on ReLU non-linearities. This branching strategy allows us to efficiently and successfully deal with high input dimensional problems with convolutional network architecture, on which previous methods fail frequently. Finally, we propose comprehensive test data sets and benchmarks which includes a collection of previously released testcases. We use the data sets to conduct a thorough experimental comparison of existing and new algorithms and to provide an inclusive analysis of the factors impacting the hardness of verification problems.

研究动机与目标

• 将神经网络验证形式化为使用 MIP/BaB 框架的全局优化问题。
• 识别现有验证方法的优点与不足，并在 BaB 下统一它们。
• 开发新的分支策略，特别是在卷积网络中的 ReLU，以提升可扩展性。
• 提出全面的数据集（包括卷积模型和合成模型），用于稳健的基准测试。
• 评估并比较现有与提出的方法，以理解验证问题中的难度因素。

提出的方法

• 将验证形式化为全局优化问题，采用一个典型表示，使其归结为检查最小值的符号。
• 通过带 big-M 约束的 MIP 对 ReLU 及其他层进行编码，并通过区间传播推导界限。
• 给出一种通用的 BaB 算法，交替进行分支、界限计算和剪枝，以在公差 epsilon 内收敛到全局最小值。
• 证明许多先前的方法（Reluplex、Planet 等）作为特殊情况适用于 BaB 框架。
• 通过改进的中间界限和凸松弛来改进界限以收紧松弛。
• 引入一种新颖的以 ReLU 为中心的分支策略，利用网络结构以处理高维和卷积架构。

实验结果

研究问题

• RQ1 如何将 PL-NNs 的验证转化为 BaB 优化问题，以及现有方法之间存在哪些共性？
• RQ2 在 BaB 中有哪些有效的分支和界限策略，可以将验证扩展到高维网络，包括卷积网络？
• RQ3 更紧的松弛和界限细化是否在验证任务中带来显著的经验加速？
• RQ4 包含卷积和合成网络的新数据集如何影响基准测试和对验证难度的理解？
• RQ5 统一的 BaB 框架是否能通过结合现有方法的优点揭示改进？

主要发现

• BaB 框架将先前的验证方法视为特殊情况，并使得它们的优点可以结合。
• 一种针对 ReLU 非线性的新分支策略在高维和卷积网络上提升了性能。
• 更紧的凸松弛和定期的中间界限细化显著加速界限计算，从而实现更快的剪枝。
• 全面数据集，包括卷积网络和合成模型，支持稳健的基准测试和对验证难度的分析。
• 该方法在基准问题上相对于先前的最先进方法实现显著加速（在某些情况下接近两个数量级）。