[Paper Review] Robustness to Adversarial Perturbations in Learning from Incomplete Data

Amir Najafi, Shin‐ichi Maeda+2 more|arXiv (Cornell University)|May 24, 2019

Adversarial Robustness in Machine LearningComputer Science33 citations

TL;DR

A semi-supervised distributionally robust learning framework (SSDRL) that unifies SSL and DRL, with an SGD-based algorithm and generalization guarantees, evaluated on MNIST, SVHN, and CIFAR-10.

ABSTRACT

What is the role of unlabeled data in an inference problem, when the presumed underlying distribution is adversarially perturbed? To provide a concrete answer to this question, this paper unifies two major learning frameworks: Semi-Supervised Learning (SSL) and Distributionally Robust Learning (DRL). We develop a generalization theory for our framework based on a number of novel complexity measures, such as an adversarial extension of Rademacher complexity and its semi-supervised analogue. Moreover, our analysis is able to quantify the role of unlabeled data in the generalization under a more general condition compared to the existing theoretical works in SSL. Based on our framework, we also present a hybrid of DRL and EM algorithms that has a guaranteed convergence rate. When implemented with deep neural networks, our method shows a comparable performance to those of the state-of-the-art on a number of real-world benchmark datasets.

Motivation & Objective

• Motivate how unlabeled data can aid learning under adversarial distribution shifts.
• Develop a framework combining semi-supervised learning with distributionally robust learning under Wasserstein ambiguity sets.
• Provide theoretical guarantees including a novel adversarial Rademacher complexity and a generalization bound.
• Propose an optimization algorithm with convergence guarantees for SSDRL.
• Demonstrate competitive empirical performance on real datasets using deep networks.

Proposed method

• Extend DRL to partially labeled data by defining a consistent distribution set over labeled and unlabeled data.
• Adopt a self-learning scheme where unlabeled data receive soft labels to avoid overfitting (optimism) or hard labels (pessimism) through a parameterized loss term.
• Define SSDRL objective as an infimum over a distribution S within a Wasserstein ball, plus a regularized entropy term for unlabeled data (Equation 5).
• Show that the inner optimization has an analytic solution via a soft-min operator (Definition 3 and Equation 6–8).
• Provide a stochastic gradient descent algorithm (Algorithm 1) with a convergence guarantee to a local minimum of the SSDRL objective (Theorem 2).
• Derive a generalization bound using a novel Semi-Supervised Monge (SSM) Rademacher complexity to handle adversarial perturbations with partial labels (Section 2.3).

Experimental results

Research questions

• RQ1 How can unlabeled data be leveraged to improve robustness under distributional adversaries in semi-supervised learning?
• RQ2 What are the theoretical generalization guarantees when combining SSL with distributionally robust learning under Wasserstein perturbations?
• RQ3 Can a practical optimization algorithm with convergence guarantees be developed for SSDRL in deep neural networks?
• RQ4 How does the proposed SSAR-based objective relate to existing DRL and SSL methods, and what are the effects of optimism vs pessimism in label assignment?

Key findings

• SSDRL integrates SSL and DRL to provide robustness against distributional perturbations in semi-supervised settings.
• The framework introduces a dual optimization with soft-labeling of unlabeled data and a Wasserstein-based adversarial risk objective.
• The method has a guaranteed convergence rate for the proposed SGD algorithm (O(T^{-1/2})) under stated conditions.
• A novel SSM Rademacher complexity offers generalization guarantees in adversarial semi-supervised learning.
• Empirical results on MNIST, SVHN, and CIFAR-10 show SSDRL matching or surpassing pseudo-labeling and supervised DRL, with competitive VAT performance on some datasets.