[论文解读] ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks

Hua Qi, Zhijie Wang+5 more|arXiv (Cornell University)|Nov 26, 2021

Adversarial Robustness in Machine Learning参考文献 63被引用 4

一句话总结

ArchRepair 通过联合优化网络架构和权重，提出了一种面向深度神经网络的块级、架构导向修复方法，以提升准确率和鲁棒性。它采用对抗感知谱分析实现精确的块定位，并在连续搜索空间中进行架构-权重联合优化，相较于现有最先进的 DNN 修复方法，在效率和有效性方面均表现更优。

ABSTRACT

Over the past few years, deep neural networks (DNNs) have achieved tremendous success and have been continuously applied in many application domains. However, during the practical deployment in the industrial tasks, DNNs are found to be erroneous-prone due to various reasons such as overfitting, lacking robustness to real-world corruptions during practical usage. To address these challenges, many recent attempts have been made to repair DNNs for version updates under practical operational contexts by updating weights (i.e., network parameters) through retraining, fine-tuning, or direct weight fixing at a neural level. In this work, as the first attempt, we initiate to repair DNNs by jointly optimizing the architecture and weights at a higher (i.e., block) level. We first perform empirical studies to investigate the limitation of whole network-level and layer-level repairing, which motivates us to explore a novel repairing direction for DNN repair at the block level. To this end, we first propose adversarial-aware spectrum analysis for vulnerable block localization that considers the neurons' status and weights' gradients in blocks during the forward and backward processes, which enables more accurate candidate block localization for repairing even under a few examples. Then, we further propose the architecture-oriented search-based repairing that relaxes the targeted block to a continuous repairing search space at higher deep feature levels. By jointly optimizing the architecture and weights in that space, we can identify a much better block architecture. We implement our proposed repairing techniques as a tool, named ArchRepair, and conduct extensive experiments to validate the proposed method. The results show that our method can not only repair but also enhance accuracy & robustness, outperforming the state-of-the-art DNN repair techniques.

研究动机与目标

• 解决现有 DNN 修复方法在神经元、层或网络级别操作时存在的局限性，这些方法通常忽略架构关系，导致计算成本高或修复效果不理想。
• 提出一种新的修复粒度——块级，以在修复效果与计算效率之间实现平衡。
• 开发一种方法，即使仅有少量故障样本，也能精确定位在特定输入下表现出异常行为的脆弱块。
• 在目标块内实现架构与权重的联合优化，以发现更优、更鲁棒的块级结构。
• 证明通过架构调整在块级别进行修复，可同时提升对自然噪声和对抗性扰动的准确率与鲁棒性。

提出的方法

• 提出对抗感知谱分析，通过在前向与反向传播过程中评估神经元可疑度和权重梯度，实现对脆弱块的定位。
• 将块级修复问题建模为基于 DARTS 的可微架构搜索问题，将块松弛为操作与特征的连续搜索空间。
• 定义一个搜索空间，其中节点代表深层特征，边代表可学习操作，从而支持基于梯度的架构与权重联合优化。
• 采用 DARTS 的稳定性改进变体（PC-DARTS），确保修复过程中架构搜索的可靠性。
• 将该方法实现为名为 ArchRepair 的工具，将块定位与架构-权重联合优化整合为统一的端到端流水线。
• 通过仅更新目标块的架构与权重来修复 DNN，最大限度减少对网络其余部分的干扰。

实验结果

研究问题

• RQ1 块级修复作为介于神经元与网络级修复之间的中间粒度，是否能比现有方法实现更高的准确率与效率？
• RQ2 即使仅有少量故障样本，如何能准确地定位 DNN 中的脆弱块？
• RQ3 在块内联合优化架构与权重，是否能相比仅优化权重的方法，带来更鲁棒、更准确的 DNN？
• RQ4 在块级别进行面向架构的修复，是否能提升对自然噪声和对抗性扰动的鲁棒性？
• RQ5 所提出的方法是否能在准确率、鲁棒性与计算成本方面超越现有最先进的 DNN 修复技术？

主要发现

• ArchRepair 即使在少样本条件下，也能通过对抗感知谱分析高精度定位脆弱块。
• 基于架构搜索的修复方法成功发现了能提升准确率与鲁棒性的改进型块架构。
• ArchRepair 在多个基准数据集上，相较于六种最先进的 DNN 修复技术，在准确率与鲁棒性方面均表现更优。
• 该方法在不重新训练整个网络的前提下，显著提升了对各类自然噪声（如高斯噪声、模糊、天气扰动）的鲁棒性。
• 通过聚焦于块级修复，ArchRepair 相较于全网络微调显著降低了计算成本，同时实现了更优的性能。
• 在块级别联合优化架构与权重，使模型对未见故障模式的泛化能力优于仅优化权重的方法。