[Paper Review] One-Shot Object Detection with Co-Attention and Co-Excitation

Ting-I Hsieh, Yi‐Chen Lo+2 more|arXiv (Cornell University)|Nov 28, 2019

Advanced Neural Network ApplicationsComputer Science116 citations

TL;DR

The paper introduces a Co-attention and Co-excitation (CoAE) framework for one-shot object detection, using non-local mutual attention and squeeze-and-co-excitation to detect unseen-class objects without fine-tuning, achieving strong results on VOC and COCO.

ABSTRACT

This paper aims to tackle the challenging problem of one-shot object detection. Given a query image patch whose class label is not included in the training data, the goal of the task is to detect all instances of the same class in a target image. To this end, we develop a novel {\em co-attention and co-excitation} (CoAE) framework that makes contributions in three key technical aspects. First, we propose to use the non-local operation to explore the co-attention embodied in each query-target pair and yield region proposals accounting for the one-shot situation. Second, we formulate a squeeze-and-co-excitation scheme that can adaptively emphasize correlated feature channels to help uncover relevant proposals and eventually the target objects. Third, we design a margin-based ranking loss for implicitly learning a metric to predict the similarity of a region proposal to the underlying query, no matter its class label is seen or unseen in training. The resulting model is therefore a two-stage detector that yields a strong baseline on both VOC and MS-COCO under one-shot setting of detecting objects from both seen and never-seen classes. Codes are available at https://github.com/timy90022/One-Shot-Object-Detection.

Motivation & Objective

• Address the challenge of detecting objects of unseen classes in a target image given a query patch from an unseen class.
• Leverage joint query-target information to produce better region proposals for one-shot detection.
• Develop a metric-learning style mechanism to rank proposals by their similarity to the query without requiring seen-class labels at test time.
• Enable class-agnostic, training-free detection of unseen objects by learning a robust similarity metric.

Proposed method

• Extend Faster R-CNN with non-local mutual attention to generate non-local region proposals that couple query and target images (co-attention).
• Introduce squeeze-and-co-excitation (SCE) to adaptively reweight feature channels from query and target for better matching (GAP, two FC/MLP layers as in SE blocks).
• Compute a query feature q from F(p) and a region feature r from F(I) after co-excitation, and use a two-layer MLP with a margin-based ranking loss to learn a similarity metric between proposals and the query.
• Train with foreground/background labeling based on IoU>0.5 and use L_CE and L_Reg losses from Faster R-CNN plus a margin-based ranking loss L_MR (with m^+ = 0.7, m^- = 0.3).
• Adopt a reduced ImageNet pre-training to avoid seeing COCO/VOC classes in backbone initialization (725 classes) and compare with full 1000-class pre-training.

Experimental results

Research questions

• RQ1 Can co-attention between a query patch and a target image improve region proposal quality for one-shot detection?
• RQ2 Does a squeeze-and-co-excitation mechanism help emphasize correlated feature channels for unseen-class detection?
• RQ3 Can a margin-based ranking loss implicitly learn a robust similarity metric between proposals and the query for both seen and unseen classes?

Key findings

• The CoAE framework yields a strong baseline for one-shot detection on VOC and COCO datasets.
• Ablation shows that both non-local (co-attention) and SCE (co-excitation) components significantly boost performance, with cumulative gains when used together.
• Margin-based ranking loss provides additional, though moderate, improvements by learning a desirable ranking of proposals.
• On COCO, Ours (1k) reaches AP50 improvements over the baseline SiamMask across splits, indicating good generalization to unseen classes.
• Visualization confirms that non-local proposals focus on target regions influenced by the query, and co-excitation reveals meaningful class-wise weight distributions (e.g., animal vs. vehicle groups).
• The method remains robust under unseen-class detection and achieves improved results over baselines on both VOC and COCO.