[Paper Review] Deep Learning for Scene Classification: A Survey

Delu Zeng, Minyu Liao+6 more|arXiv (Cornell University)|Jan 26, 2021

Advanced Image and Video Retrieval TechniquesComputer Science261 references21 citations

TL;DR

This survey provides a comprehensive overview of deep learning advancements in scene classification, covering over 200 key publications. It details CNN-based methods, scene representation strategies, RGB-D fusion techniques, and performance benchmarks, highlighting significant accuracy improvements on large-scale datasets like Places365 and SUN397 since the rise of deep learning in 2012.

ABSTRACT

Scene classification, aiming at classifying a scene image to one of the predefined scene categories by comprehending the entire image, is a longstanding, fundamental and challenging problem in computer vision. The rise of large-scale datasets, which constitute the corresponding dense sampling of diverse real-world scenes, and the renaissance of deep learning techniques, which learn powerful feature representations directly from big raw data, have been bringing remarkable progress in the field of scene representation and classification. To help researchers master needed advances in this field, the goal of this paper is to provide a comprehensive survey of recent achievements in scene classification using deep learning. More than 200 major publications are included in this survey covering different aspects of scene classification, including challenges, benchmark datasets, taxonomy, and quantitative performance comparisons of the reviewed methods. In retrospect of what has been achieved so far, this paper is also concluded with a list of promising research opportunities.

Motivation & Objective

• To provide a systematic and up-to-date review of deep learning techniques for scene classification, addressing the evolution from handcrafted features to end-to-end learning.
• To analyze and categorize over 200 major works in scene classification, focusing on challenges such as semantic ambiguity and large intra-class variations due to imaging conditions.
• To evaluate and compare the performance of different deep learning architectures and strategies, including pre-trained, fine-tuned, and specific CNN models.
• To explore emerging trends such as RGB-D scene classification and multi-modal fusion, and identify open research problems in the field.

Proposed method

• Categorizes deep learning methods for scene classification into main frameworks: pre-trained CNNs, fine-tuned CNNs, and specific CNNs, each with distinct architectures and training strategies.
• Classifies scene representation techniques into five types: global CNN features, spatially invariant features, semantic features, multi-layer features, and multi-view features.
• Reviews key strategies to improve scene representation, including encoding (e.g., Semantic FV, FCV), attention mechanisms (channel and spatial attention), contextual modeling (e.g., LSTMs, graph networks), and regularization (sparse, structured, supervised).
• Analyzes RGB-D scene classification methods that leverage depth-specific feature learning and multi-modality fusion strategies such as feature-level, consistent-feature, and distinctive-feature fusion.
• Employs a taxonomy (Figure 2) to organize methods by architecture, representation type, and strategy, enabling structured comparison across approaches.
• Uses mathematical formulations for CNN operations, including 3D convolution (Equation 1), ReLU activation (Equation 2), and Softmax cross-entropy loss (Equations 3–5), to formalize the learning process.

Experimental results

Research questions

• RQ1 How have deep learning models, particularly CNNs, transformed scene classification performance compared to traditional handcrafted methods?
• RQ2 What are the key architectural and representation strategies that have led to state-of-the-art performance in scene classification?
• RQ3 How do different scene representation techniques—global, spatially invariant, semantic, multi-layer, and multi-view—compare in terms of accuracy and robustness?
• RQ4 What role do attention mechanisms, contextual modeling, and regularization play in improving scene representation and classification accuracy?
• RQ5 How do RGB-D fusion methods enhance scene classification, and what are the most effective fusion strategies for multi-modal data?

Key findings

• Deep learning-based methods have achieved a significant performance boost on benchmark datasets like Places365 and SUN397, with top-1 accuracy improving from around 50% with handcrafted features (e.g., Dense-SIFT) to over 80% with deep CNNs.
• Pre-trained CNNs such as Places-CNN and ResNet-based models have become dominant due to their ability to generalize across diverse scene categories using ImageNet pre-training.
• Attention mechanisms, particularly channel and spatial attention, have improved feature discrimination, with methods like MFAFVNet and VSAD showing measurable gains in accuracy on challenging scene categories.
• Multi-view and multi-scale CNNs (e.g., Scale-specific CNNs, LS-DHM) outperform single-view models by capturing spatial and scale variations, especially in scenes with complex layouts.
• RGB-D fusion strategies, particularly consistent-feature and distinctive-feature based fusion, have led to improved performance on depth-aware datasets, with some models achieving over 90% top-1 accuracy on NYUv2 and ScanNet.
• Regularization techniques such as dropout, batch normalization, and multi-scale data augmentation have been critical in reducing overfitting and improving generalization in deep scene classification models.