[Paper Review] Object-Level Context Modeling For Scene Classification with Context-CNN

Syed Ashar Javed, Anil Nelakanti|arXiv (Cornell University)|May 11, 2017

Multimodal Machine Learning Applications49 references3 citations

TL;DR

This paper proposes Context-CNN, a deep learning model that enhances scene classification by modeling object-level context using object proposals and LSTM units to capture semantic relationships between objects. By integrating object-object and object-scene relationships end-to-end, the model achieves 89.03% accuracy on the LSUN validation set, demonstrating strong performance with minimal data and effective context modeling without requiring object annotations.

ABSTRACT

Convolutional Neural Networks (CNNs) have been used extensively for computer vision tasks and produce rich feature representation for objects or parts of an image. But reasoning about scenes requires integration between the low-level feature representations and the high-level semantic information. We propose a deep network architecture which models the semantic context of scenes by capturing object-level information. We use Long Short Term Memory(LSTM) units in conjunction with object proposals to incorporate object-object relationship and object-scene relationship in an end-to-end trainable manner. We evaluate our model on the LSUN dataset and achieve results comparable to the state-of-art. We further show visualization of the learned features and analyze the model with experiments to verify our model's ability to model context.

Motivation & Objective

• To address the semantic gap in scene classification by modeling high-level contextual relationships between objects.
• To improve scene classification beyond vanilla CNNs, which fail to capture complex object interactions and scene-level semantics.
• To develop an end-to-end trainable architecture that models multi-object relationships without requiring object-level annotations.
• To verify the effectiveness of LSTM-based context modeling in improving scene classification accuracy.
• To analyze the model’s feature representations and robustness through visualization and occlusion experiments.

Proposed method

• The model uses region proposal networks (RPNs) to extract object proposals from feature maps, generating candidate object bounding boxes.
• Each object proposal is processed through a RoI pooling layer to extract fixed-size deep features from a pre-trained CNN.
• Object features are fed sequentially into an LSTM network, where each time step updates the hidden state based on the current object and prior context.
• The final LSTM hidden state is used as a global image representation for scene classification via a fully connected layer and softmax.
• The entire network is trained end-to-end using cross-entropy loss with stochastic gradient descent.
• Object significance is evaluated via occlusion experiments, where bounding boxes are masked and the drop in softmax confidence is measured.

Experimental results

Research questions

• RQ1 Can modeling object-level context through LSTMs improve scene classification beyond standard CNNs?
• RQ2 Does the sequential processing of object proposals via LSTM enhance the model’s ability to capture semantic relationships between objects?
• RQ3 How does the order of object input affect the final classification score, and which objects are most discriminative for scene categories?
• RQ4 To what extent does the model rely on specific objects or spatial configurations for accurate scene prediction?
• RQ5 Can the model achieve state-of-the-art performance without requiring object-level annotations?

Key findings

• The Context-CNN model achieves 89.03% top-1 accuracy on the LSUN validation set, placing it among the top-performing models on this benchmark.
• The model converges with only 200k images from the 10M-image LSUN dataset, indicating high data efficiency.
• Occlusion of the first LSTM input (highest objectness score) causes the largest drop in accuracy—63.3% when the first box is obscured—highlighting its critical role in classification.
• The model effectively learns to distinguish scenes based on characteristic objects (e.g., bed for bedroom, sofa for living room), even when appearance and pose vary.
• Visualizations show that LSTM features become increasingly discriminative over time steps, confirming the model’s capacity to build contextual representations.
• Control experiments confirm that the LSTM component is essential, as removing it leads to significant performance degradation.