[Paper Review] A Discriminative Representation of Convolutional Features for Indoor Scene Recognition

Salman Khan, Munawar Hayat+3 more|arXiv (Cornell University)|Jun 16, 2015

Advanced Image and Video Retrieval TechniquesComputer Science58 references115 citations

TL;DR

This paper proposes a novel discriminative mid-level feature representation for indoor scene recognition by transforming structured convolutional activations into a more discriminative space using multiple codebooks of scene representative patches (SRPs). By combining supervised SRPs from a new large-scale indoor object dataset (1,300 categories) with unsupervised SRPs, and employing max-margin hyperplane encoding, the method achieves state-of-the-art performance on five major scene classification benchmarks, significantly outperforming prior approaches in handling intra-class variability and inter-class similarity.

ABSTRACT

Indoor scene recognition is a multi-faceted and challenging problem due to the diverse intra-class variations and the confusing inter-class similarities. This paper presents a novel approach which exploits rich mid-level convolutional features to categorize indoor scenes. Traditionally used convolutional features preserve the global spatial structure, which is a desirable property for general object recognition. However, we argue that this structuredness is not much helpful when we have large variations in scene layouts, e.g., in indoor scenes. We propose to transform the structured convolutional activations to another highly discriminative feature space. The representation in the transformed space not only incorporates the discriminative aspects of the target dataset, but it also encodes the features in terms of the general object categories that are present in indoor scenes. To this end, we introduce a new large-scale dataset of 1300 object categories which are commonly present in indoor scenes. Our proposed approach achieves a significant performance boost over previous state of the art approaches on five major scene classification datasets.

Motivation & Objective

• To address the challenge of high intra-class variability and inter-class similarity in indoor scene recognition.
• To improve upon traditional global or local feature representations by introducing mid-level features that preserve both local details and structural relationships.
• To develop a robust feature encoding scheme that enhances discriminative power for complex indoor scenes.
• To create and release the first large-scale dataset of 1,300 indoor object categories relevant to scene recognition.
• To demonstrate superior performance on diverse, challenging indoor scene classification datasets.

Proposed method

• Extract dense, uniform image patches from deep convolutional features to form mid-level representations.
• Construct multiple smaller codebooks of Scene Representative Patches (SRPs), combining supervised SRPs from a new 1,300-category indoor object dataset with unsupervised SRPs from training data.
• Encode features using max-margin hyperplanes to model discriminative associations between patches and scene categories.
• Apply max-pooling across multiple codebooks to aggregate discriminative patch responses for final representation.
• Use sparse coding and classifier similarity metrics for feature encoding, with ablation studies to validate component effectiveness.
• Leverage data augmentation and patch contribution heat maps to improve robustness and interpretability.

Experimental results

Research questions

• RQ1 Can mid-level representations derived from convolutional features outperform global or local representations in indoor scene recognition?
• RQ2 Does combining supervised and unsupervised SRPs in multiple codebooks improve classification accuracy compared to single codebooks?
• RQ3 How effective is max-margin hyperplane encoding in capturing discriminative patch relationships for scene categorization?
• RQ4 To what extent does the proposed method generalize to datasets with high clutter, scale, and pose variations?
• RQ5 Can a large-scale, semantically labeled indoor object dataset enhance scene recognition performance?

Key findings

• The proposed DUCA method achieved 98.7% accuracy on the UIUC 8-Sports dataset, a 10.2% improvement over the previous best method.
• On the Graz-02 dataset, the method reached 98.6% accuracy, a 12.6% improvement over the prior state of the art.
• The combination of supervised and unsupervised codebooks yielded 71.8% accuracy on the MIT-67 dataset, outperforming individual components and single large codebooks.
• Multiple smaller codebooks reduced feature encoding time by ~20 seconds per image compared to a single large codebook, while maintaining or improving performance.
• Max-pooling across codebooks produced the best results, with 71.8% accuracy on MIT-67, compared to 69.7% with mean-pooling.
• Heat maps revealed that the most discriminative mid-level patches contributed most to correct predictions, confirming the method's focus on salient scene elements.