[Paper Review] Spatial Reasoning for Few-Shot Object Detection

Geonuk Kim, Hong-Gyu Jung+1 more|arXiv (Cornell University)|Nov 2, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a few-shot object detection framework, FSOD-SR, that enhances feature representation of novel categories by modeling spatial relationships among region proposals (RoIs) using a graph convolutional network (GCN). By encoding geometric and co-occurrence relationships between novel and base category RoIs, and employing a novel spatial data augmentation strategy, the method achieves state-of-the-art performance on PASCAL VOC and MS COCO, outperforming prior methods by significant margins in few-shot settings.

ABSTRACT

Although modern object detectors rely heavily on a significant amount of training data, humans can easily detect novel objects using a few training examples. The mechanism of the human visual system is to interpret spatial relationships among various objects and this process enables us to exploit contextual information by considering the co-occurrence of objects. Thus, we propose a spatial reasoning framework that detects novel objects with only a few training examples in a context. We infer geometric relatedness between novel and base RoIs (Region-of-Interests) to enhance the feature representation of novel categories using an object detector well trained on base categories. We employ a graph convolutional network as the RoIs and their relatedness are defined as nodes and edges, respectively. Furthermore, we present spatial data augmentation to overcome the few-shot environment where all objects and bounding boxes in an image are resized randomly. Using the PASCAL VOC and MS COCO datasets, we demonstrate that the proposed method significantly outperforms the state-of-the-art methods and verify its efficacy through extensive ablation studies.

Motivation & Objective

• To address the limitation of existing few-shot object detectors that rely solely on local RoI features without leveraging contextual spatial relationships.
• To improve feature representation for novel categories by modeling co-occurrence and geometric relationships among objects in an image.
• To develop a spatial data augmentation strategy that increases training diversity by resizing RoIs simultaneously, enhancing spatial reasoning robustness.
• To achieve strong few-shot generalization performance on both in-domain and cross-dataset settings.
• To mitigate performance degradation on base categories during few-shot fine-tuning through dynamic feature fusion.

Proposed method

• A spatial graph is constructed where nodes represent RoIs (from RPN) and edges encode geometric and semantic relatedness between RoIs, particularly between novel and base categories.
• Edge weights are computed based on IoU (intersection over union) of bounding boxes and classification scores from base categories, capturing spatial and semantic relatedness.
• A graph convolutional network (GCN) propagates and aggregates contextual information across the spatial graph to refine RoI features.
• Enhanced features are formed by concatenating original RoI features with GCN-processed contextual features, improving discriminative power for novel categories.
• A novel spatial data augmentation strategy resizes all objects in an image simultaneously T times, generating exponentially more training samples for diverse scale and spatial configuration learning.
• The method integrates these components into a meta-learning framework for few-shot fine-tuning, enabling detection of novel categories with only a few examples.

Figure 1: Conceptual example of the human visual system for recognizing new objects using a spatial reasoning process. Even if a human has little experience of new objects and has trouble recognizing the objects individually, it is possible to recognize the objects applying a spatial reasoning proce

Experimental results

Research questions

• RQ1 Can modeling spatial relationships between novel and base category objects improve few-shot object detection performance?
• RQ2 How does incorporating geometric and co-occurrence patterns among objects affect feature representation for rare categories?
• RQ3 Can a graph-based spatial reasoning mechanism outperform conventional RoI-level feature manipulation in few-shot detection?
• RQ4 Does spatial data augmentation that resizes all objects simultaneously improve generalization in low-data regimes?
• RQ5 Can the proposed method maintain strong performance on base categories while significantly boosting novel category detection?

Key findings

• The proposed FSOD-SR method achieves 43.2 mAP on the 20-way 10-shot PASCAL VOC benchmark, outperforming the previous state-of-the-art method (MPSR) by 0.9 mAP.
• On the cross-dataset generalization setting (MS COCO base, PASCAL VOC novel), FSOD-SR achieves 43.2 mAP, surpassing MPSR (42.3 mAP) and Meta R-CNN (37.4 mAP).
• Abllation studies confirm that both the spatial graph and spatial data augmentation contribute significantly to performance gains, with the GCN-based reasoning improving mAP by up to 3.5 points.
• Visualization shows that the model learns meaningful spatial relationships, connecting novel objects (e.g., mouse) to semantically plausible base objects (e.g., monitor, keyboard) based on spatial proximity and co-occurrence.
• The method demonstrates strong generalization to unseen domains, achieving 44.6 mAP on 20-way 1-shot MS COCO when trained on 60 base categories from MS COCO and fine-tuned on PASCAL VOC.
• Despite performance gains on novel categories, the method shows a slight drop in base category mAP compared to TFA, indicating a trade-off that future work may address via dynamic feature fusion.

Figure 3: Detailed flowchart of the edge regression. We transform the RoI visual features and the corresponding coordinates from an RPN into latent spaces to calculate the relatedness of RoIs. Finally, the ReLU function is exploited to build a sparse graph.