[Paper Review] ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks

Hua Qi, Zhijie Wang+5 more|arXiv (Cornell University)|Nov 26, 2021

Adversarial Robustness in Machine Learning63 references4 citations

TL;DR

ArchRepair proposes block-level, architecture-oriented repair for deep neural networks by jointly optimizing network architecture and weights to improve accuracy and robustness. It uses adversarial-aware spectrum analysis for precise block localization and a continuous search space for architecture-weight co-optimization, outperforming state-of-the-art DNN repair methods in both efficiency and effectiveness.

ABSTRACT

Over the past few years, deep neural networks (DNNs) have achieved tremendous success and have been continuously applied in many application domains. However, during the practical deployment in the industrial tasks, DNNs are found to be erroneous-prone due to various reasons such as overfitting, lacking robustness to real-world corruptions during practical usage. To address these challenges, many recent attempts have been made to repair DNNs for version updates under practical operational contexts by updating weights (i.e., network parameters) through retraining, fine-tuning, or direct weight fixing at a neural level. In this work, as the first attempt, we initiate to repair DNNs by jointly optimizing the architecture and weights at a higher (i.e., block) level. We first perform empirical studies to investigate the limitation of whole network-level and layer-level repairing, which motivates us to explore a novel repairing direction for DNN repair at the block level. To this end, we first propose adversarial-aware spectrum analysis for vulnerable block localization that considers the neurons' status and weights' gradients in blocks during the forward and backward processes, which enables more accurate candidate block localization for repairing even under a few examples. Then, we further propose the architecture-oriented search-based repairing that relaxes the targeted block to a continuous repairing search space at higher deep feature levels. By jointly optimizing the architecture and weights in that space, we can identify a much better block architecture. We implement our proposed repairing techniques as a tool, named ArchRepair, and conduct extensive experiments to validate the proposed method. The results show that our method can not only repair but also enhance accuracy & robustness, outperforming the state-of-the-art DNN repair techniques.

Motivation & Objective

• Address the limitations of existing DNN repair methods that operate at the neuron, layer, or network level, which often neglect architectural relationships and lead to high computational costs or suboptimal fixes.
• Identify a new repair granularity—block-level—that offers a balanced trade-off between repair effectiveness and computational efficiency.
• Develop a method to precisely localize vulnerable blocks in a DNN that exhibit faulty behavior under specific inputs, even with few examples.
• Enable joint optimization of both architecture and weights within a targeted block to discover improved, robust block-level structures.
• Demonstrate that repairing at the block level with architectural changes can enhance both accuracy and robustness against natural and adversarial corruptions.

Proposed method

• Propose adversarial-aware spectrum analysis to localize vulnerable blocks by evaluating neuron suspiciousness and weight gradients during forward and backward passes.
• Formulate the block-level repair problem as a differentiable architecture search (DARTS-based) problem, relaxing the block to a continuous search space of operations and features.
• Define a search space where nodes represent deep features and edges represent learnable operations, enabling gradient-based joint optimization of architecture and weights.
• Use a stability-improved variant of DARTS (PC-DARTS) to ensure reliable architecture search during repair.
• Implement the method as a tool called ArchRepair, which integrates block localization and architecture-weight co-optimization in a unified pipeline.
• Apply the method to repair DNNs by updating only the targeted block’s architecture and weights, minimizing interference with the rest of the network.

Experimental results

Research questions

• RQ1 Can block-level repair, as a middle-ground granularity between neuron and network-level repair, achieve better accuracy and efficiency than existing methods?
• RQ2 How can vulnerable blocks be accurately localized in a DNN even when only a few failure examples are available?
• RQ3 Can joint optimization of architecture and weights within a block lead to more robust and accurate DNNs compared to weight-only repair?
• RQ4 Does architecture-oriented repair at the block level improve robustness against natural corruptions and adversarial perturbations?
• RQ5 Can the proposed method outperform state-of-the-art DNN repair techniques in terms of accuracy, robustness, and computational cost?

Key findings

• ArchRepair localizes vulnerable blocks with high precision using adversarial-aware spectrum analysis, even under few-shot conditions.
• The architecture-oriented search-based repair method successfully discovers improved block architectures that enhance both accuracy and robustness.
• ArchRepair outperforms six state-of-the-art DNN repair techniques in terms of accuracy and robustness across multiple benchmark datasets.
• The method achieves significant improvements in robustness against various natural corruptions (e.g., Gaussian noise, blur, and weather corruptions) without retraining the entire network.
• By focusing on block-level repair, ArchRepair reduces computational cost compared to full-network retraining while achieving superior performance.
• The joint optimization of architecture and weights at the block level enables the model to generalize better to unseen failure patterns than weight-only repair methods.