[Paper Review] Interventional Few-Shot Learning

Zhongqi Yue, Hanwang Zhang+2 more|arXiv (Cornell University)|Sep 28, 2020

Domain Adaptation and Few-Shot LearningComputer Science77 references94 citations

TL;DR

IFSL treats pre-trained knowledge as a confounder in few-shot learning and uses backdoor adjustment to intervene, yielding state-of-the-art 1-shot and 5-shot results across mini ImageNet, tiered ImageNet, and cross-domain CUB.

ABSTRACT

We uncover an ever-overlooked deficiency in the prevailing Few-Shot Learning (FSL) methods: the pre-trained knowledge is indeed a confounder that limits the performance. This finding is rooted from our causal assumption: a Structural Causal Model (SCM) for the causalities among the pre-trained knowledge, sample features, and labels. Thanks to it, we propose a novel FSL paradigm: Interventional Few-Shot Learning (IFSL). Specifically, we develop three effective IFSL algorithmic implementations based on the backdoor adjustment, which is essentially a causal intervention towards the SCM of many-shot learning: the upper-bound of FSL in a causal view. It is worth noting that the contribution of IFSL is orthogonal to existing fine-tuning and meta-learning based FSL methods, hence IFSL can improve all of them, achieving a new 1-/5-shot state-of-the-art on extit{mini}ImageNet, extit{tiered}ImageNet, and cross-domain CUB. Code is released at https://github.com/yue-zhongqi/ifsl.

Motivation & Objective

• Identify and formalize the deficiency where pre-training acts as a confounder in FSL.
• Propose a causal framework (SCM) for FSL to justify interventions.
• Develop three practical IFSL implementations based on backdoor adjustment.
• Demonstrate that IFSL is orthogonal to and improves existing fine-tuning and meta-learning FSL methods.
• Provide empirical evidence of improved 1-/5-shot performance on standard benchmarks and analyze performance across similarity between support and query sets.

Proposed method

• Formulate a Structural Causal Model (SCM) to represent the causal relations among pre-trained knowledge, sample features, and labels.
• Apply backdoor adjustment to estimate P(Y|do(X)) as a causal intervention.
• Present three practical IFSL implementations: feature-wise adjustment, class-wise adjustment, and a combined adjustment.
• Leverage NWGM (Normalized Weighted Geometric Mean) to approximate the class-wise component for efficiency.
• Ensure IFSL can be plugged into existing FSL baselines without altering their core training objectives.
• Provide algorithmic details and equations in the appendices for exact instantiations across backbones and classifiers.

Experimental results

Research questions

• RQ1 Can pre-trained knowledge act as a confounder in few-shot learning, biasing P(Y|X) away from true causal effects?
• RQ2 How can backdoor adjustment be instantiated practically for FSL to approximate P(Y|do(X)) without many-shot data?
• RQ3 Do feature-wise, class-wise, or combined adjustments improve FSL performance, and are these improvements orthogonal to fine-tuning and meta-learning methods?
• RQ4 What are the empirical gains of IFSL on standard benchmarks (mini ImageNet, tiered ImageNet) and cross-domain tasks (CUB) across 1-shot and 5-shot settings?
• RQ5 How does IFSL influence model attention (CAM-Acc) and robustness across query hardness?

Key findings

• IFSL yields consistent accuracy gains when plugged into both fine-tuning and meta-learning baselines across 1-/5-shot settings.
• IFSL achieves new state-of-the-art results on mini ImageNet and tiered ImageNet in 1-/5-shot scenarios.
• IFSL provides improvements in cross-domain generalization (mini ImageNet to CUB) over linear classifiers and remains beneficial for inductive as well as transductive baselines.
• The gains are larger in 1-shot settings, indicating higher susceptibility to confounding bias with fewer examples.
• IFSL improves attention to the true object as evidenced by CAM-Acc visualizations, suggesting reliance on correct visual semantics rather than confounded cues.
• IFSL is orthogonal to existing FSL methods, improving baselines without requiring changes to their core training procedures.
• Across varying similarity between S (support) and Q (query), IFSL improves performance in all regimes, including harder query samples.