[论文解读] Distributional Robustness with IPMs and links to Regularization and GANs

Hisham Husain|arXiv (Cornell University)|Jun 1, 2020

Adversarial Robustness in Machine Learning被引用 6

一句话总结

本文建立了基于积分概率度量（IPMs）的分布鲁棒优化（DRO）与机器学习中正则化的统一联系。研究表明，任意IPM下的DRO对应于一类正则化惩罚，恢复并扩展了关于MMD和Wasserstein距离的结果，同时将GAN目标与分布鲁棒性联系起来。

ABSTRACT

Robustness to adversarial attacks is an important concern due to the fragility of deep neural networks to small perturbations and has received an abundance of attention in recent years. Distributionally Robust Optimization (DRO), a particularly promising way of addressing this challenge, studies robustness via divergence-based uncertainty sets and has provided valuable insights into robustification strategies such as regularization. In the context of machine learning, the majority of existing results have chosen $f$-divergences, Wasserstein distances and more recently, the Maximum Mean Discrepancy (MMD) to construct uncertainty sets. We extend this line of work for the purposes of understanding robustness via regularization by studying uncertainty sets constructed with Integral Probability Metrics (IPMs) - a large family of divergences including the MMD, Total Variation and Wasserstein distances. Our main result shows that DRO under extit{any} choice of IPM corresponds to a family of regularization penalties, which recover and improve upon existing results in the setting of MMD and Wasserstein distances. Due to the generality of our result, we show that other choices of IPMs correspond to other commonly used penalties in machine learning. Furthermore, we extend our results to shed light on adversarial generative modelling via $f$-GANs, constituting the first study of distributional robustness for the $f$-GAN objective. Our results unveil the inductive properties of the discriminator set with regards to robustness, allowing us to give positive comments for several penalty-based GAN methods such as Wasserstein-, MMD- and Sobolev-GANs. In summary, our results intimately link GANs to distributional robustness, extend previous results on DRO and contribute to our understanding of the link between regularization and robustness at large.

研究动机与目标

• 理解分布鲁棒优化（DRO）与机器学习中正则化之间的联系。
• 通过使用更广泛的积分概率度量（IPMs）类，将现有的DRO框架扩展至f-散度和Wasserstein距离之外。
• 揭示不同IPMs如何对应于机器学习中的已知正则化惩罚。
• 通过分布鲁棒性的视角，为f-GANs的对抗生成建模提供理论基础。
• 以鲁棒性与泛化性为框架，解释GAN中判别器集合的归纳偏置。

提出的方法

• 作者使用由积分概率度量（IPMs）定义的不确定性集来构建DRO，该类度量包括MMD、总变差和Wasserstein距离。
• 他们推导了任意IPM下DRO问题的对偶表示，表明鲁棒目标对应于带惩罚项的正则化经验风险最小化。
• 该惩罚项被证明等价于MMD情形下的再生核希尔伯特空间（RKHS）中的范数，以及Wasserstein距离情形下的利普希茨约束。
• 通过将判别器的优化解释为IPM下的DRO形式，将该框架扩展至分析f-GANs，揭示其诱导鲁棒性的特性。
• 理论分析表明，多种GAN变体——Wasserstein-、MMD-和Sobolev-GANs——可被解释为使用特定IPM求解DRO问题。
• 该方法为正则化与鲁棒性提供了统一视角，表明不同IPMs在模型中诱导出不同的归纳偏置。

实验结果

研究问题

• RQ1 在IPM下的分布鲁棒优化如何与机器学习中的正则化相关联？
• RQ2 使用不同IPM的DRO公式自然引出哪些已知的正则化惩罚？
• RQ3 f-GAN目标能否被解释为一种分布鲁棒性形式？
• RQ4 不同GAN判别器架构在鲁棒性方面诱导了哪些归纳偏置？
• RQ5 MMD、Wasserstein和Sobolev GANs如何与各自IPM下的DRO相关联？

主要发现

• 任意IPM下的DRO对应于带惩罚项的正则化经验风险最小化问题，该惩罚项源自IPM的对偶表示。
• 该框架恢复并推广了现有关于MMD和Wasserstein DRO的结果，表明它们是更广泛IPM正则化方案中的特例。
• 机器学习中常用的其他惩罚项，如Sobolev GAN中的惩罚项，被证明源于特定IPM的选择。
• f-GAN目标通过基于IPM的不确定性集，被正式关联到分布鲁棒性，为GAN中的对抗训练提供了新解释。
• 分析表明，f-GAN中判别器集合通过约束模型对IPM所捕捉的分布偏移保持鲁棒，从而诱导出鲁棒性。
• 结果为基于惩罚的GAN方法（如Wasserstein-和MMD-GANs）的成功提供了理论依据，将其与鲁棒优化联系起来。