[Paper Review] Semantic Understanding of Scenes through the ADE20K Dataset

Bolei Zhou, Hang Zhao+5 more|arXiv (Cornell University)|Aug 18, 2016

Advanced Neural Network Applications34 references190 citations

TL;DR

Introduces the ADE20K densely annotated dataset with pixel-wise labels for scenes, objects, and parts, and provides baselines for scene parsing and instance segmentation, plus analysis of batch normalization effects.

ABSTRACT

Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision. Despite the community's efforts in data collection, there are still few image datasets covering a wide range of scenes and object categories with dense and detailed annotations for scene parsing. In this paper, we introduce and analyze the ADE20K dataset, spanning diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. A generic network design called Cascade Segmentation Module is then proposed to enable the segmentation networks to parse a scene into stuff, objects, and object parts in a cascade. We evaluate the proposed module integrated within two existing semantic segmentation networks, yielding significant improvements for scene parsing. We further show that the scene parsing networks trained on ADE20K can be applied to a wide variety of scenes and objects.

Motivation & Objective

• Create a large, densely annotated image dataset covering diverse scenes, objects, and object parts with open vocabulary naming.
• Provide pixel-wise benchmarks for scene parsing and instance segmentation built on ADE20K.
• Evaluate baseline segmentation models and re-implement state-of-the-art methods for open-source accessibility.
• Analyze the impact of synchronized batch normalization and other training settings on segmentation performance.
• Explore joint training of object and part annotations and discuss potential applications to hierarchical scene understanding.

Proposed method

• Annotate 20,210 training, 2,000 validation, and 3,000 test images with dense object, part, and attribute labels via a single expert annotator.
• Construct SceneParse150 and InstSeg100 benchmarks from ADE20K for pixel-wise scene parsing and instance segmentation, respectively.
• Re-implement and release open-source PyTorch baselines for scene parsing (including DilatedResNet variants, PSPNet, UPerNet) and evaluate on SceneParse150.
• Experiment with batch normalization settings (synchronized vs unsynchronized vs frozen BN) and batch sizes to assess effects on segmentation accuracy.
• Train instance segmentation using Mask R-CNN with FPN-50 backbone on InstSeg100 and analyze multi-scale training effects.

Experimental results

Research questions

• RQ1 How does ADE20K’s dense annotation enable broader scene understanding including object parts and parts of parts?
• RQ2 What are effective baseline architectures and training settings for pixel-wise scene parsing on SceneParse150?
• RQ3 How does synchronized batch normalization influence segmentation performance on ADE20K?
• RQ4 What is the baseline performance for instance segmentation on InstSeg100 using Mask R-CNN, and how does multi-scale training impact it?
• RQ5 What insights can be gained from the object-part relationships for scene understanding and knowledge base linking?

Key findings

• SceneParse150 baselines show DilatedVGG and DilatedResNet variants achieve higher mean IoU than FCN/SegNet, with cascaded versions providing further gains.
• Re-implemented state-of-the-art models (e.g., PSPNet, UPerNet) in PyTorch show 3-7 percentage points higher mean IoU over plain baselines, highlighting the importance of context.
• Synchronized BN with a batch size of 16 yields higher Pixel Acc. and Mean IoU than smaller or unsynchronized BN settings, with BN size around 4 being a practical inflection point.
• InstSeg100 results indicate Mask R-CNN multi-scale training substantially improves mean mAP (e.g., from 0.1832 to 0.2241 overall), while small objects remain challenging.
• ADE20K’s dense annotations reveal that most images contain multiple objects (average ~19.5 instances, ~10.5 classes) and that over 76% of object instances have parts (avg ~3 parts per object).