[논문 리뷰] Rectifying the Shortcut Learning of Background for Few-Shot Learning

Xu Luo, Longhui Wei외 5명|arXiv (Cornell University)|2021. 07. 16.

Domain Adaptation and Few-Shot Learning참고 문헌 69인용 수 45

한 줄 요약

이 논문은 이미지 배경을 소수 샷 학습에서 해로운 단축 지식으로 식별하고, 대비 학습과 클러스터링을 활용해 학습 및 평가를 위해 전경(foreground)을 추출하는 COSOC를 제안하여 소수 샷 성능을 개선한다.

ABSTRACT

The category gap between training and evaluation has been characterised as one of the main obstacles to the success of Few-Shot Learning (FSL). In this paper, we for the first time empirically identify image background, common in realistic images, as a shortcut knowledge helpful for in-class classification but ungeneralizable beyond training categories in FSL. A novel framework, COSOC, is designed to tackle this problem by extracting foreground objects in images at both training and evaluation without any extra supervision. Extensive experiments carried on inductive FSL tasks demonstrate the effectiveness of our approaches.

연구 동기 및 목표

• 이미지 배경이 소수 샷 학습에서 단축 지식으로 작용하는지 여부와 이것이 훈련 및 평가 범주 간 일반화에 어떤 영향을 미치는지 식별한다.
• 추가 감독 없이도 모델의 주의를 전경 객체에 집중시키는 학습-평가 프레임워크(COSOC)를 제안한다.
• 표준 FSL 벤치마크(mini ImageNet 및 tiered ImageNet)에서 전경 중심 전략의 효과를 입증한다.
• 전경 기반 학습/평가를 바람직한 기준(솔리시티/saliency)과 비교하고, 전경 식별에서 대비 학습의 역할을 분석한다.]
• method:[" COS: 대비 전처리 후 임의 패치를 크롭하고, 클래스별로 특징을 클러스터링하며 대표 전경 패치를 선택하고 학습 중 융합 샘플링을 위한 전경 점수를 할당한다."," SOC: 지원 집합 내에서 이미지 간의 공유 콘텐츠를 찾아 클래스 표현을 형성하고 평가 시 전경 패치에 대해 패치 수준 특징 매칭을 수행하는 공유 객체 농축기이다."," Fusion sampling: 원본 이미지와 전경 크롭의 혼합 학습으로 배경 감소와 복합적 장면에 대한 적응성 간의 균형을 맞춘다."," 전경 추출은 바운딩 박스 없이 전경 객체에 속할 가능성이 높은 패치를 식별하기 위해 대조 학습(Exemplar)을 사용한다."," 평가에서는 패치 간 매칭을 반복적으로 수행하여 전경 콘텐츠에 집중하고 클래스 점수를 패치 유사도의 가중 합으로 계산한다."," Saliency-baseline과의 비교에서 COSOC의 전경 초점은 일반적인 주목도(generic saliency)보다는 공유된 클래스 콘텐츠에 의해 주도된다.]
• research_questions:[

제안 방법

• COS: clustering-based object seeker that, after contrastive pretraining, crops random patches, clusters their features per class, selects representative foreground patches, and assigns foreground scores for fusion sampling during training.
• SOC: shared object concentrator that, within a support set, finds shared content across images to form class representations and performs patch-level feature matching to foreground patches during evaluation.
• Fusion sampling: trains on a mix of original and foreground crops to balance background reduction with adaptability to complex scenes.
• Foreground extraction uses contrastive learning (Exemplar) to identify patches likely belonging to foreground objects without bounding boxes.
• Evaluation uses iterative feature matching across crops to concentrate on foreground content and compute class scores as a weighted sum of patch similarities.
• Comparison with saliency baselines shows COSOC’s foreground focus is driven by shared class content rather than generic saliency.

실험 결과

연구 질문

• RQ1 Does image background act as a harmful shortcut in few-shot learning, and how does foreground vs background influence training and evaluation performance?
• RQ2 Can a foreground-focused framework (COSOC) improve few-shot generalization without extra supervision, and how does it compare with saliency-based foreground extraction?
• RQ3 What is the role of contrastive learning in foreground identification and subsequent FSL performance?
• RQ4 How does fusion sampling of foreground and original images affect trade-offs between robustness to complex scenes and background shortcut mitigation?
• RQ5 How close can SOC-based evaluation approach the upper bound set by using ground-truth foreground during evaluation?

주요 결과

• Background acts as a harmful shortcut for FSL, degrading evaluation performance when training includes background information.
• Removing background at evaluation significantly improves performance across models and datasets; training with foreground-only data generally yields better generalization to new classes.
• Fusion sampling of foreground and original images at training provides a balance, achieving competitive results on both original and foreground-only evaluations.
• COS improves foreground discrimination via contrastive learning; SOC leverages shared inter-image information to concentrate on foreground during evaluation, approaching upper-bound performance
• COSOC-based methods outperform saliency-based foreground extractors across 1-shot and 5-shot settings on mini ImageNet and tiered ImageNet.
• The SOC component is crucial for maximizing gains; using COS alone without SOC yields weaker improvements.