[Paper Review] Object-to-Scene: Learning to Transfer Object Knowledge to Indoor Scene Recognition

Bo Miao, Liguang Zhou+3 more|arXiv (Cornell University)|Aug 1, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes Object-to-Scene (OTS), a one-stream framework that enhances indoor scene recognition by extracting high-level object features via a segmentation network and an Object Feature Aggregation Module (OFAM), then modeling object relations using an Object Attention Module (OAM) and Global Relation Aggregation Module (GRAM). OTS achieves a 2%+ accuracy gain over state-of-the-art methods without additional streams, demonstrating superior performance through explicit object-level representation and relation learning.

ABSTRACT

Accurate perception of the surrounding scene is helpful for robots to make reasonable judgments and behaviours. Therefore, developing effective scene representation and recognition methods are of significant importance in robotics. Currently, a large body of research focuses on developing novel auxiliary features and networks to improve indoor scene recognition ability. However, few of them focus on directly constructing object features and relations for indoor scene recognition. In this paper, we analyze the weaknesses of current methods and propose an Object-to-Scene (OTS) method, which extracts object features and learns object relations to recognize indoor scenes. The proposed OTS first extracts object features based on the segmentation network and the proposed object feature aggregation module (OFAM). Afterwards, the object relations are calculated and the scene representation is constructed based on the proposed object attention module (OAM) and global relation aggregation module (GRAM). The final results in this work show that OTS successfully extracts object features and learns object relations from the segmentation network. Moreover, OTS outperforms the state-of-the-art methods by more than 2\% on indoor scene recognition without using any additional streams. Code is publicly available at: https://github.com/FreeformRobotics/OTS.

Motivation & Objective

• To address the limitations of current indoor scene recognition methods that fail to effectively utilize object-level features and relations.
• To develop a unified, one-stream framework that directly transfers knowledge from object-level features to scene-level recognition.
• To improve scene representation by modeling long-range object relations using an attention-based mechanism.
• To achieve higher accuracy than existing methods without relying on multi-stream architectures or handcrafted features.

Proposed method

• The method uses a segmentation network combined with an Object Feature Aggregation Module (OFAM) to extract high-level, semantic-rich object features from input images.
• An Object Attention Module (OAM) is proposed to implicitly learn long-range dependencies and co-occurring relations between detected objects using a flexible attention mechanism.
• The Global Relation Aggregation Module (GRAM) fuses object features and relations using depthwise and pointwise convolutions to generate a high-level scene representation vector.
• A fully connected layer performs final scene classification based on the aggregated scene representation.
• The entire framework is end-to-end trainable and operates as a single-stream model, avoiding the need for additional feature streams.
• The model is trained using cross-entropy loss and evaluated on standard indoor scene benchmarks.

Experimental results

Research questions

• RQ1 Can explicit modeling of object features and their relations improve indoor scene recognition beyond traditional feature-based or global feature-based methods?
• RQ2 How effective is a one-stream architecture that leverages segmentation-based object features without adding extra streams?
• RQ3 To what extent can an attention-based module like OAM capture long-range object relations for better scene representation?
• RQ4 How does GRAM compare to conventional fusion methods like fully connected layers or pooling in terms of accuracy and efficiency?

Key findings

• OTS achieves 85.9% top-1 accuracy on the indoor scene recognition benchmark, outperforming state-of-the-art methods by more than 2% without using any additional streams.
• The ablation study shows that two object attention blocks (OAB) yield the best performance (85.9%), while adding more blocks degrades accuracy due to model complexity.
• The proposed Object Attention Block achieves 85.9% accuracy with only 1.2M parameters and 211.5M FLOPs, outperforming self-attention (84.7%) and non-local (82.3%) modules.
• GRAM achieves the highest accuracy (85.9%) with minimal parameters (2.3M) and FLOPs (2.3M), significantly outperforming fully connected layers and pooling layers in both accuracy and efficiency.
• Failure cases are primarily due to object detection errors—e.g., misclassifying a kitchen as a bathroom when a pot is misdetected as a toilet—indicating that performance is sensitive to segmentation quality.
• The backbone features from the segmentation network alone are sufficient for strong scene recognition, suggesting that additional streams are not necessary when proper object-level feature learning is applied.