[论文解读] Adversarial Fault Tolerant Training for Deep Neural Networks.

Vasisht Duddu, Dushyant Rao+1 more|arXiv (Cornell University)|Jul 6, 2019

Adversarial Robustness in Machine Learning参考文献 43被引用 7

一句话总结

本文通过解耦特征提取与分类任务，提出一种基于多准则目标的对抗性容错训练方法：先使用两个生成对抗网络（重建与分布平滑）进行无监督预训练，再进行有监督微调。该方法在FashionMNIST和CIFAR10数据集上实现了高准确率，并展现出对固定为0故障的优越容错能力，优于标准正则化方法。

ABSTRACT

Deep Learning Accelerators are prone to faults which manifest in the form of errors in Neural Networks. Fault Tolerance in Neural Networks is crucial in real-time safety critical applications requiring computation for long durations. Neural Networks with high regularisation exhibit superior fault tolerance, however, at the cost of classification accuracy. In the view of difference in functionality, a Neural Network is modelled as two separate networks, i.e, the Feature Extractor with unsupervised learning objective and the Classifier with a supervised learning objective. Traditional approaches of training the entire network using a single supervised learning objective is insufficient to achieve the objectives of the individual components optimally. In this work, a novel multi-criteria objective function, combining unsupervised training of the Feature Extractor followed by supervised tuning with Classifier Network is proposed. The unsupervised training solves two games simultaneously in the presence of adversary neural networks with conflicting objectives to the Feature Extractor. The first game minimises the loss in reconstructing the input image for indistinguishability given the features from the Extractor, in the presence of a generative decoder. The second game solves a minimax constraint optimisation for distributional smoothening of feature space to match a prior distribution, in the presence of a Discriminator network. The resultant strongly regularised Feature Extractor is combined with the Classifier Network for supervised fine-tuning. The proposed Adversarial Fault Tolerant Neural Network Training is scalable to large networks and is independent of the architecture. The evaluation on benchmarking datasets: FashionMNIST and CIFAR10, indicates that the resultant networks have high accuracy with superior tolerance to stuck at 0 faults compared to widely used regularisers.

研究动机与目标

• 解决安全关键应用中深度神经网络在容错性与分类准确率之间的权衡问题。
• 通过解耦特征提取与分类训练目标，在不牺牲模型准确率的前提下提升容错能力。
• 开发一种可扩展、与网络架构无关的训练框架，以增强对固定为0故障的鲁棒性。
• 将特征提取器建模为生成器与判别器之间的双人极小极大博弈，以强化鲁棒性与泛化能力。
• 在基准数据集上证明该方法相比传统正则化方法具有更优的容错性能。

提出的方法

• 将神经网络分解为无监督的特征提取器与有监督的分类器，并分别进行训练。
• 使用两个同时进行的生成对抗网络训练特征提取器：通过生成器实现重建，通过判别器实现分布平滑。
• 使用重建损失确保从提取的特征中能够准确重建输入图像。
• 应用极小极大损失使特征分布匹配先验分布，从而提升泛化能力与鲁棒性。
• 使用标注数据对联合的特征提取器与分类器进行标准有监督学习微调。
• 将对抗性训练与强正则化相结合，以在不损害准确率的前提下增强容错能力。

实验结果

研究问题

• RQ1 解耦特征提取与分类训练是否能提升深度神经网络的容错能力？
• RQ2 采用双生成对抗目标（重建与分布平滑）的对抗性训练是否能增强对固定为0故障的鲁棒性？
• RQ3 所提方法是否能在显著提升容错能力的同时保持高分类准确率，优于标准正则化方法？
• RQ4 所提训练框架是否可在不同深度学习模型中实现可扩展性与架构无关性？
• RQ5 在故障条件下，对抗性容错训练在FashionMNIST和CIFAR10等标准基准数据集上的表现如何？

主要发现

• 所提方法在FashionMNIST和CIFAR10上均实现了高于标准正则化方法的分类准确率，同时展现出更优的容错能力。
• 对特征提取器进行双重对抗性训练显著提升了对固定为0故障的鲁棒性，优于传统正则化技术。
• 通过重建与分布平滑进行的无监督预训练，生成了高度正则化的特征空间，使其在故障条件下仍能良好泛化。
• 该方法具备可扩展性且与网络架构无关，适用于大规模深度神经网络。
• 评估结果证实，对抗性训练后的网络即使在持续故障条件下仍能保持高性能，验证了其在安全关键系统中的适用性。
• 对抗性训练与有监督微调的结合，使模型在准确率与容错能力之间实现了比端到端有监督训练更优的平衡。