[论文解读] A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness

James Diffenderfer, Brian R. Bartoldson+3 more|arXiv (Cornell University)|Jun 16, 2021

Generative Adversarial Networks and Image Synthesis被引用 12

一句话总结

该论文表明，彩票剪枝（lottery ticket-style pruning）与量化技术可生成紧凑、准确且鲁棒的（CARD）深度神经网络——令人惊讶的是，其在分布外（OOD）鲁棒性方面甚至优于更大的模型。通过训练稀疏的、二值权重的CARD网络，并采用基于谱相似性的领域自适应CARD-Deck框架，该方法在CIFAR-10-C上实现了SOTA性能（标准准确率96.8%，鲁棒准确率92.75%），在CIFAR-100-C上也实现了SOTA性能（标准准确率80.6%，鲁棒准确率71.3%），且内存占用更低。

ABSTRACT

Successful adoption of deep learning (DL) in the wild requires models to be: (1) compact, (2) accurate, and (3) robust to distributional shifts. Unfortunately, efforts towards simultaneously meeting these requirements have mostly been unsuccessful. This raises an important question: Is the inability to create Compact, Accurate, and Robust Deep neural networks (CARDs) fundamental? To answer this question, we perform a large-scale analysis of popular model compression techniques which uncovers several intriguing patterns. Notably, in contrast to traditional pruning approaches (e.g., fine tuning and gradual magnitude pruning), we find that "lottery ticket-style" approaches can surprisingly be used to produce CARDs, including binary-weight CARDs. Specifically, we are able to create extremely compact CARDs that, compared to their larger counterparts, have similar test accuracy and matching (or better) robustness -- simply by pruning and (optionally) quantizing. Leveraging the compactness of CARDs, we develop a simple domain-adaptive test-time ensembling approach (CARD-Decks) that uses a gating module to dynamically select appropriate CARDs from the CARD-Deck based on their spectral-similarity with test samples. The proposed approach builds a "winning hand'' of CARDs that establishes a new state-of-the-art (on RobustBench) on CIFAR-10-C accuracies (i.e., 96.8% standard and 92.75% robust) and CIFAR-100-C accuracies (80.6% standard and 71.3% robust) with better memory usage than non-compressed baselines (pretrained CARDs and CARD-Decks available at https://github.com/RobustBench/robustbench). Finally, we provide theoretical support for our empirical findings.

研究动机与目标

• 探究尽管先前在整合这些特性方面存在失败，紧凑、准确且鲁棒的（CARD）深度神经网络是否仍可实现。
• 确定模型压缩技术——尤其是彩票剪枝风格的剪枝——是否能提升而非损害OOD鲁棒性。
• 开发一种利用CARD紧凑性的动态集成策略，以在分布偏移下实现更优鲁棒性。
• 为压缩带来的鲁棒性提升提供理论支持。

提出的方法

• 应用彩票剪枝风格的剪枝与量化技术，训练稀疏的、二值权重的神经网络，以实现高准确率与鲁棒性。
• 提出CARD-Deck框架：一种基于测试样本与CARD之间谱相似性进行门控选择的动态CARD集成。
• 利用测试样本与CARD之间的谱相似性，在推理阶段引导自适应模型选择。
• 在训练过程中使用数据增强技术（如AugMix、高斯噪声）以提升单个CARD的泛化能力与鲁棒性。
• 通过条件Wasserstein距离的理论分析，界定集成设置下的OOD偏移与泛化误差。
• 推导出一种鲁棒性分解，将经验鲁棒性、泛化差距与分布偏移分量分离。

实验结果

研究问题

• RQ1 能否通过模型压缩构建出紧凑、准确且鲁棒的深度神经网络（CARD）？
• RQ2 彩票剪枝风格的剪枝与量化是否能提升而非降低分布外鲁棒性？
• RQ3 现有鲁棒性技术（如数据增强）能否与CARD有效结合以进一步提升性能？
• RQ4 CARD的紧凑性是否能支持高效、自适应的集成策略，使其优于标准集成？
• RQ5 是否存在理论基础解释压缩带来的鲁棒性增益？

主要发现

• 彩票剪枝风格的剪枝与量化可生成CARD，其OOD鲁棒性与大尺寸全精度模型相当或更优，同时保持高测试准确率。
• 所提出的CARD-Deck框架在CIFAR-10-C上实现了新的SOTA性能，标准准确率达96.8%，鲁棒准确率达92.75%，优于非压缩基线模型。
• 在CIFAR-100-C上，CARD-Deck实现了80.6%的标准准确率与71.3%的鲁棒准确率，再次以更低内存占用创下SOTA纪录。
• CARD-Deck中基于谱相似性的门控机制能持续为每类扰动类型选择表现最佳的CARD，显著提升在多样化分布偏移下的鲁棒性。
• 理论分析证实，压缩带来的鲁棒性增益源于有界的泛化误差与分布偏移误差，支持了实证发现。
• 本研究验证了CARD假设：鲁棒且紧凑的网络在训练初期即已存在，可通过彩票剪枝初始化识别，从而将彩票剪枝原理扩展至鲁棒性领域。