[Paper Review] Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Chenxi Liu, Liang-Chieh Chen+5 more|arXiv (Cornell University)|Jan 10, 2019

Advanced Neural Network Applications93 references149 citations

TL;DR

Auto-DeepLab introduces hierarchical neural architecture search that jointly optimizes network-level and cell-level structures for semantic segmentation, achieving strong results without ImageNet pretraining and efficient search (~3 GPU days).

ABSTRACT

Recently, Neural Architecture Search (NAS) has successfully identified neural network architectures that exceed human designed ones on large-scale image classification. In this paper, we study NAS for semantic image segmentation. Existing works often focus on searching the repeatable cell structure, while hand-designing the outer network structure that controls the spatial resolution changes. This choice simplifies the search space, but becomes increasingly problematic for dense image prediction which exhibits a lot more network level architectural variations. Therefore, we propose to search the network level structure in addition to the cell level structure, which forms a hierarchical architecture search space. We present a network level search space that includes many popular designs, and develop a formulation that allows efficient gradient-based architecture search (3 P100 GPU days on Cityscapes images). We demonstrate the effectiveness of the proposed method on the challenging Cityscapes, PASCAL VOC 2012, and ADE20K datasets. Auto-DeepLab, our architecture searched specifically for semantic image segmentation, attains state-of-the-art performance without any ImageNet pretraining.

Motivation & Objective

• Extend neural architecture search from image classification to dense semantic segmentation.
• Propose a two-level hierarchical search space that includes network-level and cell-level architectures.
• Develop a differentiable, gradient-based NAS framework to efficiently search the hierarchy.
• Demonstrate strong segmentation performance without ImageNet pretraining and compare to state-of-the-art baselines.

Proposed method

• Define a two-level hierarchical search space: a network-level trellis for spatial resolution changes and a cell-level DAG describing layer operations.
• Use a differentiable continuous relaxation of architectures with alphas for cell-level operations and betas for network-level transitions.
• Optimize architecture parameters and network weights alternately using gradient-based updates on split training data (trainA/trainB).
• Decode discrete architectures using greedy decoding for cells (top predecessors and argmax operators) and Viterbi decoding for the network path.
• Attach an Atrous Spatial Pyramid Pooling (ASPP) module to each resolution during search with a simplified multi-branch setup.
• Train from scratch on Cityscapes with 321x321 crops and evaluate on Cityscapes, PASCAL VOC 2012, and ADE20K.

Experimental results

Research questions

• RQ1 Can neural architecture search be effectively extended to dense image prediction tasks like semantic segmentation?
• RQ2 Does jointly searching network-level and cell-level architectures yield better performance than searching only cells?
• RQ3 How efficient can differentiable NAS be for high-resolution, dense prediction tasks?
• RQ4 How do Auto-DeepLab variants perform without ImageNet pretraining across Cityscapes, VOC 2012, and ADE20K?

Key findings

• Without ImageNet pretraining, Auto-DeepLab-L significantly outperforms FRRN-B by 8.6% and GridNet by 10.9% on Cityscapes test set.
• Auto-DeepLab matches the performance of DeepLabv3+ (with pretraining) while being 2.23x faster in Multi-Adds.
• A lightweight Auto-DeepLab-S achieves 80.9% on Cityscapes test with substantially fewer parameters (10.15M) and 333.25B Multi-Adds.
• Auto-DeepLab-L reaches 82.1% on Cityscapes test with coarse annotations and 55.2% fewer Multi-Adds than DeepLabv3+; without pretraining, the best model outperforms several baselines on Cityscapes.
• On PASCAL VOC 2012 and ADE20K, the best Auto-DeepLab variants outperform several state-of-the-art models trained with limited pretraining; VOC test reaches up to 85.6% mIOU with ImageNet/COCO pretraining variants.
• The proposed differentiable two-level NAS is roughly 1000x faster than prior dense-prediction NAS methods (e.g., DPC) and finds architectures that generalize across multiple datasets.