[Paper Review] Places: An Image Database for Deep Scene Understanding

Bolei Zhou, Aditya Khosla+3 more|arXiv (Cornell University)|Oct 6, 2016

Advanced Image and Video Retrieval TechniquesComputer Science31 references175 citations

TL;DR

The paper introduces Places, a 10M-image scene-centric database spanning 476 categories, built via multi-stage crowdsourcing and bootstrapping, and demonstrates strong CNN-based scene classification performance. It also compares scene-centric versus object-centric features and provides benchmarks and visualization insights.

ABSTRACT

The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification at tasks such as object and scene recognition. Here we describe the Places Database, a repository of 10 million scene photographs, labeled with scene semantic categories and attributes, comprising a quasi-exhaustive list of the types of environments encountered in the world. Using state of the art Convolutional Neural Networks, we provide impressive baseline performances at scene classification. With its high-coverage and high-diversity of exemplars, the Places Database offers an ecosystem to guide future progress on currently intractable visual recognition problems.

Motivation & Objective

• Motivate the creation of a large-scale, diverse, and category-rich scene dataset to advance deep scene understanding.
• Describe the construction pipeline combining web data collection, crowdsourced labeling, and semi-automatic bootstrapping.
• Establish benchmarks (Places365 variants, Places205, Places88) to enable fair evaluation of scene recognition methods.
• Explore the effectiveness of scene-centric CNN features (Places-CNN) versus object-centric features (ImageNet-CNN) for scene classification.
• Provide qualitative analyses and visualizations to understand learned representations in scene-centric networks.

Proposed method

• Aggregate 10 million images from the web using SUN-derived scene categories and adjective-based queries to increase diversity.
• Crowdsourced labeling via Amazon Mechanical Turk to select true exemplars for 476 scene categories across multiple rounds of validation.
• Semi-automatic bootstrapping with a CNN (AlexNet) to classify remaining unlabeled images and guide targeted manual annotation.
• Merge and disambiguate near-synonymous categories and refine labels to improve category separability.
• Train and evaluate CNN baselines (AlexNet, GoogLeNet, VGG, and ResNet variants) on Places205 and Places365 subsets; compare against ImageNet-CNN features.
• Analyze feature representations and provide visualization of units receptive fields and synthetic inputs to interpret learned scene concepts.

Experimental results

Research questions

• RQ1 How large and diverse must a scene-centric dataset be to enable robust deep scene understanding?
• RQ2 Can crowdsourcing combined with bootstrapping reliably create a high-coverage Places dataset from web images?
• RQ3 How do scene-centric CNN features (Places-CNN) compare to object-centric features (ImageNet-CNN) on scene-centric benchmarks?
• RQ4 What benchmarks best represent progress in scene recognition, and how do different CNN architectures perform on them?
• RQ5 What do the internal units of Places-CNNs reveal about learned scene representations, and how can visualization aid interpretation?

Key findings

• Places: 10,624,928 images across 434 place categories, built via a multi-step process with crowdsourced validation and bootstrapping.
• Places365-Standard contains 1,803,460 training images; Places365-Challenge adds ~8 million training images; Places205 has 2.5 million images across 205 categories.
• Places-CNN features outperform ImageNet-CNN features on scene-centric tasks, with Places365-VGG achieving 63.24% Top-1 on SUN397, and hybrid 1365-VGG achieving best average across eight datasets.
• On Places205 and SUN205, Places-CNNs (e.g., Places205-VGG, Places205-GoogLeNet) significantly surpass the ImageNet-CNN baselines in Top-1/Top-5 accuracy.
• The unified Places benchmarks (Places365-Standard/Challenge, Places205, Places88) enable consistent evaluation and progress tracking for scene recognition research.
• Visualization shows Places-CNN units detect scene parts (bed, chair, buildings) rather than object parts, highlighting a distinct learned representation from object-centric networks.