[Paper Review] Adversarially Robust Few-Shot Learning: A Meta-Learning Approach

Micah Goldblum, Liam Fowl+1 more|arXiv (Cornell University)|Oct 2, 2019

Adversarial Robustness in Machine Learning33 references42 citations

TL;DR

The paper introduces Adversarial Querying (AQ), an algorithm-agnostic meta-learning method to produce few-shot models that are robust to adversarial attacks, outperforming robust transfer learning on standard benchmarks.

ABSTRACT

Previous work on adversarially robust neural networks for image classification requires large training sets and computationally expensive training procedures. On the other hand, few-shot learning methods are highly vulnerable to adversarial examples. The goal of our work is to produce networks which both perform well at few-shot classification tasks and are simultaneously robust to adversarial examples. We develop an algorithm, called Adversarial Querying (AQ), for producing adversarially robust meta-learners, and we thoroughly investigate the causes for adversarial vulnerability. Moreover, our method achieves far superior robust performance on few-shot image classification tasks, such as Mini-ImageNet and CIFAR-FS, than robust transfer learning.

Motivation & Objective

• Motivate the need for robustness in few-shot learning settings.
• Develop a meta-learning framework that yields adversarially robust feature extractors for few-shot tasks.
• Demonstrate that adversarial querying (AQ) yields superior robustness and clean accuracy compared to existing defenses.
• Show that keeping the backbone fixed and only fine-tuning the last layer improves robustness.
• Provide algorithm-agnostic evidence across multiple meta-learning backbones and datasets.

Proposed method

• Formulate a minimax objective for meta-learning: min_theta E_{S,(x,y)} [ max_{||δ||_p<ε} L(F_{A(θ,S)}, x+δ, y) ].
• Propose adversarial querying (AQ): alternate between fine-tuning on support data and adversarially perturbing query data to maximize loss, backpropagating through A.
• Attack only query data during AQ to balance robustness gains with computational efficiency.
• Test AQ across ProtoNet, R2-D2, and MetaOptNet on CIFAR-FS and Mini-ImageNet.
• Compare AQ to adversarially trained transfer learning and other defenses to show robustness gains.
• Examine the impact of backbone vs. head choice on robustness and show that using a fixed backbone with last-layer retraining improves performance.

Experimental results

Research questions

• RQ1 Can adversarial querying produce meta-learners that are robust to strong adversarial attacks in few-shot settings?
• RQ2 Is AQ algorithm-agnostic across meta-learning backbones, and does robustness depend more on the head or backbone?
• RQ3 How does AQ compare to adversarial training and pre-processing defenses in few-shot robustness?
• RQ4 What is the effect of only fine-tuning the last layer on natural and robust accuracy?
• RQ5 Do robust feature extractors learned via AQ transfer better to robustness than robustly trained transfer learning?

Key findings

• AQ-based meta-learners achieve higher robust accuracy and competitive natural accuracy than robust transfer learning on Mini-ImageNet and CIFAR-FS.
• Naturally trained meta-learners are highly vulnerable to 20-step PGD attacks, while AQ significantly improves robustness across multiple backbones.
• Attacking only query data (not support data) during AQ provides robust gains with lower computational cost than perturbing support data.
• Re-training only the final layer during fine-tuning yields better natural and robust accuracy than updating the entire network for AQ.
• The R2-D2 head contributes markedly to robustness, more so than the embedding, across AQ models.
• AQ outperforms adversarially trained transfer learning in robustness across tested datasets and backbones.