[Paper Review] VC Classes are Adversarially Robustly Learnable, but Only Improperly

Omar Montasser, Steve Hanneke+1 more|arXiv (Cornell University)|Feb 12, 2019

Adversarial Robustness in Machine LearningComputer Science21 references43 citations

TL;DR

The paper shows that any finite VC-dimension class is robustly PAC learnable under adversarial perturbations using improper learning rules, while some finite-VC classes are not robustly PAC learnable with any proper rule.

ABSTRACT

We study the question of learning an adversarially robust predictor. We show that any hypothesis class $\mathcal{H}$ with finite VC dimension is robustly PAC learnable with an improper learning rule. The requirement of being improper is necessary as we exhibit examples of hypothesis classes $\mathcal{H}$ with finite VC dimension that are not robustly PAC learnable with any proper learning rule.

Motivation & Objective

• Characterize when adversarially robust PAC learning is possible for a given hypothesis class under a specified adversary.
• Distinguish between proper and improper learning in robust settings and identify when improper learning is necessary.
• Provide finite-sample and asymptotic bounds on the sample complexity for robust learning under VC assumptions.

Proposed method

• Formulate robust risk under an adversary mapping U and define robust PAC learning in realizable and agnostic settings.
• Show a negative result: there exists a finite-VC class that is not robustly PAC learnable with any proper rule.
• Prove a positive result: any finite-VC class is robustly PAC learnable via an improper learning rule, with sample complexity bounds tied to dual VC dimension vc*(H).
• Develop a sample-compression/boosting based learning scheme that achieves zero empirical robust loss and yields a robust learner.
• Use a reduction from agnostic to realizable learning to extend realizable results to the agnostic setting.

Experimental results

Research questions

• RQ1 Can finite VC-dimension hypothesis classes be robustly PAC learned under adversarial perturbations, and does proper learning suffice?
• RQ2 What are the limitations of proper learning for robust PAC learning, and can improper learning overcome them?
• RQ3 How do sample complexity and bounds depend on VC(H) and the dual VC dimension vc*(H) under adversarial robustness?
• RQ4 Can reductions from agnostic to realizable learning translate to robust learning settings?
• RQ5 What constructive learning procedures can achieve robust generalization for finite-VC classes?

Key findings

• There exists a finite-VC class with vc(H) ≤ 1 that is not robustly PAC learnable with respect to some adversary under any proper learning rule.
• For any adversary U and any finite-VC class H, there exists an improper learning rule that robustly PAC learns H, albeit with potential exponential dependence on vc(H).
• Finite VC dimension is sufficient for improper robust learnability, enabling robust generalization via RERM-like approaches extended with compression/boosting techniques.
• A realizable robust learnability bound is obtained using a sample-compression approach; the bound depends on vc(H) and the dual VC dimension vc*(H).
• In the agnostic setting, robust learnability follows from a reduction to the realizable case, yielding finite-sample guarantees with bounds that scale with vc(H) and vc*(H).
• The compression-based argument provides constructive procedures and implies that robust learnability may require improper learning in some cases.