[Paper Review] Rectifying the Shortcut Learning of Background for Few-Shot Learning

Xu Luo, Longhui Wei+5 more|arXiv (Cornell University)|Jul 16, 2021

Domain Adaptation and Few-Shot LearningComputer Science69 references45 citations

TL;DR

The paper identifies image background as harmful shortcut knowledge in few-shot learning and proposes COSOC, a foreground-focused framework that uses contrastive learning and clustering to extract foregrounds for training and evaluation, yielding improved few-shot performance.

ABSTRACT

The category gap between training and evaluation has been characterised as one of the main obstacles to the success of Few-Shot Learning (FSL). In this paper, we for the first time empirically identify image background, common in realistic images, as a shortcut knowledge helpful for in-class classification but ungeneralizable beyond training categories in FSL. A novel framework, COSOC, is designed to tackle this problem by extracting foreground objects in images at both training and evaluation without any extra supervision. Extensive experiments carried on inductive FSL tasks demonstrate the effectiveness of our approaches.

Motivation & Objective

• Identify whether image background acts as shortcut knowledge in few-shot learning and how it affects generalization across training and evaluation categories.
• Propose a training-evaluation framework (COSOC) that concentrates model attention on foreground objects without extra supervision.
• Demonstrate the effectiveness of foreground-focused strategies on standard FSL benchmarks (mini ImageNet and tiered ImageNet).
• Compare foreground-based training/evaluation with saliency baselines and analyze the role of contrastive learning in foreground identification.

Proposed method

• COS: clustering-based object seeker that, after contrastive pretraining, crops random patches, clusters their features per class, selects representative foreground patches, and assigns foreground scores for fusion sampling during training.
• SOC: shared object concentrator that, within a support set, finds shared content across images to form class representations and performs patch-level feature matching to foreground patches during evaluation.
• Fusion sampling: trains on a mix of original and foreground crops to balance background reduction with adaptability to complex scenes.
• Foreground extraction uses contrastive learning (Exemplar) to identify patches likely belonging to foreground objects without bounding boxes.
• Evaluation uses iterative feature matching across crops to concentrate on foreground content and compute class scores as a weighted sum of patch similarities.
• Comparison with saliency baselines shows COSOC’s foreground focus is driven by shared class content rather than generic saliency.

Experimental results

Research questions

• RQ1 Does image background act as a harmful shortcut in few-shot learning, and how does foreground vs background influence training and evaluation performance?
• RQ2 Can a foreground-focused framework (COSOC) improve few-shot generalization without extra supervision, and how does it compare with saliency-based foreground extraction?
• RQ3 What is the role of contrastive learning in foreground identification and subsequent FSL performance?
• RQ4 How does fusion sampling of foreground and original images affect trade-offs between robustness to complex scenes and background shortcut mitigation?
• RQ5 How close can SOC-based evaluation approach the upper bound set by using ground-truth foreground during evaluation?

Key findings

• Background acts as a harmful shortcut for FSL, degrading evaluation performance when training includes background information.
• Removing background at evaluation significantly improves performance across models and datasets; training with foreground-only data generally yields better generalization to new classes.
• Fusion sampling of foreground and original images at training provides a balance, achieving competitive results on both original and foreground-only evaluations.
• COS improves foreground discrimination via contrastive learning; SOC leverages shared inter-image information to concentrate on foreground during evaluation, approaching upper-bound performance
• COSOC-based methods outperform saliency-based foreground extractors across 1-shot and 5-shot settings on mini ImageNet and tiered ImageNet.
• The SOC component is crucial for maximizing gains; using COS alone without SOC yields weaker improvements.