[论文解读] Adversarially Robust Low Dimensional Representations

Pranjal Awasthi, Vaggos Chatziafratis+2 more|arXiv (Cornell University)|Nov 29, 2019

Adversarial Robustness in Machine Learning参考文献 88被引用 5

一句话总结

本文提出了一种鲁棒主成分分析（PCA）的变体，确保在对抗性扰动（特别是 $ℓ_q$ 范数，如 $ℓ_\infty$）下，低维数据表示依然保持准确与稳定。该方法提出了一种多项式时间算法，对测试时和训练时的鲁棒性均具有常数因子近似保证，从而在对抗性扰动环境下实现了高效且可靠的降维。

ABSTRACT

Many machine learning systems are vulnerable to small perturbations made to inputs either at test time or at training time. This has received much recent interest on the empirical front due to applications where reliability and security are critical. However, theoretical understanding of algorithms that are robust to adversarial perturbations is limited. In this work we focus on Principal Component Analysis (PCA), a ubiquitous algorithmic primitive in machine learning. We formulate a natural robust variant of PCA where the goal is to find a low dimensional subspace to represent the given data with minimum projection error, that is in addition robust to small perturbations measured in $\ell_q$ norm (say $q=\infty$). Unlike PCA which is solvable in polynomial time, our formulation is computationally intractable to optimize as it captures a variant of the well-studied sparse PCA objective as a special case. We show the following results: -Polynomial time algorithm that is constant factor competitive in the worst-case with respect to the best subspace, in terms of the projection error and the robustness criterion. -We show that our algorithmic techniques can also be made robust to adversarial training-time perturbations, in addition to yielding representations that are robust to adversarial perturbations at test time. Specifically, we design algorithms for a strong notion of training-time perturbations, where every point is adversarially perturbed up to a specified amount. -We illustrate the broad applicability of our algorithmic techniques in addressing robustness to adversarial perturbations, both at training time and test time. In particular, our adversarially robust PCA primitive leads to computationally efficient and robust algorithms for both unsupervised and supervised learning problems such as clustering and learning adversarially robust classifiers.

研究动机与目标

• 解决当前对主成分分析（PCA）等基础机器学习原语中对抗性鲁棒性的理论理解不足的问题。
• 制定一种鲁棒PCA变体，最小化投影误差，同时在测试时对小的 $ℓ_q$-范数扰动具有鲁棒性。
• 将鲁棒性扩展至对抗性训练时扰动，即数据点在模型训练前即被污染。
• 开发计算高效的算法，在对抗性条件下仍保持强理论保证。
• 展示鲁棒PCA在下游任务（如聚类和鲁棒分类）中的广泛适用性。

提出的方法

• 定义一种鲁棒PCA目标，通过最小化在 $ℓ_q$-有界扰动下的最坏情况投影误差，利用算子范数 $\|\Pi\|_{q\to 2}$ 捕获对抗性鲁棒性。
• 提出一种多项式时间算法，对最优鲁棒子空间的投影误差和鲁棒性准则均实现常数因子近似。
• 引入一种新框架，通过建模每个数据点最多受指定 $ℓ_q$-范数预算的扰动，来处理训练时的对抗性扰动。
• 利用算子范数和谱性质的近似算法，界定最坏情况下的鲁棒性误差。
• 通过基于投影的特征变换，将鲁棒PCA原语应用于无监督任务（如聚类）和有监督任务（如学习鲁棒分类器）。
• 使用集中不等式和矩阵扰动界等理论工具，分析对抗性污染下均值估计的稳定性。

实验结果

研究问题

• RQ1 我们能否设计一种多项式时间算法，为对抗性 $ℓ_q$-范数扰动下的鲁棒PCA公式提供常数因子近似保证？
• RQ2 如何将鲁棒性从测试时扰动扩展到对抗性训练时污染，即数据在学习前即被扰动？
• RQ3 在分类任务中使用鲁棒低维投影时，自然准确率与对抗性鲁棒性之间的权衡是什么？
• RQ4 鲁棒PCA在多大程度上可作为基础原语，用于提升聚类和分类等下游学习任务的鲁棒性？
• RQ5 在对抗性 $ℓ_q$-范数污染下，均值估计的信息论极限是什么？其与投影矩阵鲁棒性的关系如何？

主要发现

• 所提算法在对抗性测试时扰动下，对Frobenius范数和谱范数误差均实现了对最优鲁棒子空间的常数因子近似。
• 该方法通过确保投影矩阵在 $ℓ_q$-有界数据污染下保持稳定，从而保证了对抗性训练时扰动的鲁棒性。
• 对于对抗性污染下的均值估计，论文建立了误差上界 $O\left((1 + \kappa\delta/\sigma)\max(\sigma, \sqrt{\sigma\|\mu\|})\right)$，其中 $\kappa = \|\Pi\|_{q\to 2}$，$\delta$ 为扰动半径，$\sigma$ 为数据方差。
• 该算法对均值估计实现了相对误差保证 $O\left((1 + \kappa\delta/\sigma)\max(\sigma/\|\mu\|, \sqrt{\sigma/\|\mu\|})\right)$，表明即使真实均值稀疏，仍具有鲁棒性。
• 鲁棒PCA原语可实现高效且鲁棒的聚类与分类，且在两种场景下均具有对抗性鲁棒性的理论保证。
• 论文建立了计算下界，表明精确优化鲁棒PCA目标是不可行的，因其推广了稀疏PCA。