[Paper Review] Rectifying the Shortcut Learning of Background: Shared Object Concentration for Few-Shot Image Recognition.

Xu Luo, Longhui Wei+5 more|arXiv (Cornell University)|Jul 16, 2021

Domain Adaptation and Few-Shot Learning32 references4 citations

TL;DR

This paper proposes COSOC, a few-shot learning framework that mitigates shortcut learning from image backgrounds by identifying and focusing on shared foreground objects across images. Using contrastive feature clustering and a fusion sampling strategy during pretraining, and foreground-aware feature matching during inference, COSOC achieves state-of-the-art performance on few-shot image recognition benchmarks by suppressing background bias.

ABSTRACT

Few-Shot image classification aims to utilize pretrained knowledge learned from a large-scale dataset to tackle a series of downstream classification tasks. Typically, each task involves only few training examples from brand-new categories. This requires the pretraining models to focus on well-generalizable knowledge, but ignore domain-specific information. In this paper, we observe that image background serves as a source of domain-specific knowledge, which is a shortcut for models to learn in the source dataset, but is harmful when adapting to brand-new classes. To prevent the model from learning this shortcut knowledge, we propose COSOC, a novel Few-Shot Learning framework, to automatically figure out foreground objects at both pretraining and evaluation stage. COSOC is a two-stage algorithm motivated by the observation that foreground objects from different images within the same class share more similar patterns than backgrounds. At the pretraining stage, for each class, we cluster contrastive-pretrained features of randomly cropped image patches, such that crops containing only foreground objects can be identified by a single cluster. We then force the pretraining model to focus on found foreground objects by a fusion sampling strategy; at the evaluation stage, among images in each training class of any few-shot task, we seek for shared contents and filter out background. The recognized foreground objects of each class are used to match foreground of testing images. Extensive experiments tailored to inductive FSL tasks on two benchmarks demonstrate the state-of-the-art performance of our method.

Motivation & Objective

• To address the problem of models learning spurious correlations from image backgrounds during pretraining, which harms generalization to new few-shot classes.
• To identify and focus on foreground objects that are shared across images of the same class, reducing reliance on domain-specific background features.
• To develop a two-stage framework that operates during both pretraining and inference to detect and filter out background-dominated image patches.
• To improve few-shot generalization by ensuring models learn well-generalizable, object-centric representations instead of background-conditional patterns.

Proposed method

• At pretraining, random crops of image patches are embedded using a contrastive-pretrained backbone, and their features are clustered to identify a dominant cluster corresponding to foreground objects.
• A fusion sampling strategy is applied to prioritize training samples from the identified foreground cluster, encouraging the model to focus on object-centric features.
• During inference, for each few-shot support set, shared content across images is detected using feature clustering to isolate foreground regions.
• Foreground features from support images are used to match against query image features, filtering out background-irrelevant signals.
• The method leverages self-supervised contrastive learning to extract discriminative features before clustering, enabling robust object detection without annotations.
• The framework operates in a weakly supervised manner, requiring no bounding boxes or segmentation masks, making it scalable and applicable to real-world few-shot tasks.

Experimental results

Research questions

• RQ1 Can foreground object patterns be reliably identified across images of the same class without human annotations?
• RQ2 Does suppressing background features during pretraining improve few-shot generalization performance?
• RQ3 Can a two-stage approach—clustering during pretraining and foreground matching during inference—outperform standard few-shot learning methods?
• RQ4 How effective is the fusion sampling strategy in focusing the model on foreground objects during pretraining?
• RQ5 To what extent does shared object concentration reduce shortcut learning from background artifacts?

Key findings

• COSOC achieves state-of-the-art performance on two standard few-shot image recognition benchmarks, outperforming existing methods.
• The method significantly reduces model reliance on background features by identifying and emphasizing shared foreground patterns across images of the same class.
• The fusion sampling strategy during pretraining leads to more robust feature learning, as evidenced by improved zero-shot generalization and fine-tuning accuracy.
• Foreground clustering at inference time effectively filters out background noise, improving matching accuracy on query images.
• The approach generalizes well across diverse few-shot tasks without requiring additional annotations or architectural modifications.
• The ablation study confirms that background suppression via foreground clustering is a key factor in the model's improved performance.