[Paper Review] Searching for Efficient Multi-Scale Architectures for Dense Image Prediction

Liang-Chieh Chen, Maxwell D. Collins+6 more|arXiv (Cornell University)|Sep 11, 2018

Domain Adaptation and Few-Shot Learning339 citations

TL;DR

This paper applies neural architecture search to dense image prediction, introducing Dense Prediction Cells (DPCs) and a fast proxy task to discover architectures that outperform hand-crafted models on Cityscapes, PASCAL-Person-Part, and PASCAL VOC 2012.

ABSTRACT

The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning algorithms. Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks. An open question is the degree to which such methods may generalize to new domains. In this work we explore the construction of meta-learning techniques for dense image prediction focused on the tasks of scene parsing, person-part segmentation, and semantic image segmentation. Constructing viable search spaces in this domain is challenging because of the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a survey of techniques in dense image prediction, we construct a recursive search space and demonstrate that even with efficient random search, we can identify architectures that outperform human-invented architectures and achieve state-of-the-art performance on three dense prediction tasks including 82.7\% on Cityscapes (street scene parsing), 71.3\% on PASCAL-Person-Part (person-part segmentation), and 87.9\% on PASCAL VOC 2012 (semantic image segmentation). Additionally, the resulting architecture is more computationally efficient, requiring half the parameters and half the computational cost as previous state of the art systems.

Motivation & Objective

• Motivate automatic architecture design for dense image prediction beyond image classification.
• Construct a tractable search space that captures multi-scale context for high-resolution imagery.
• Develop a fast proxy task to evaluate candidate architectures without full large-scale training.
• Demonstrate that random search within the proposed space can outperform human-designed architectures on multiple dense prediction tasks.

Proposed method

• Define Dense Prediction Cell (DPC) as a DAG-based, multi-branch module that concatenates branch outputs to form multi-scale context.
• Specify each branch input as either the backbone feature map or earlier branch outputs, enabling recursive, multi-scale connections.
• Describe operator options including 1x1 convolutions, 3x3 atrous separable convolutions with varied rates, and spatial pyramid pooling with different grid sizes.
• Aggregate branch outputs via concatenation to form the DPC output, enabling parallel and cascaded context aggregation.
• Construct a large but tractable search space: with B=5 branches, total configurations approximate B! × 81^B.
• Utilize a black-box optimization approach (Vizier) with random search to identify high-performing architectures under a given evaluation budget.

Experimental results

Research questions

• RQ1 Can meta-learning / neural architecture search discover compact, high-performing multi-scale architectures for dense prediction tasks?
• RQ2 Does a carefully designed recursive multi-scale search space (DPC) plus a fast proxy task yield architectures that surpass human-designed models on scene labeling and related tasks?
• RQ3 How do the discovered DPC architectures compare in accuracy and efficiency to existing state-of-the-art methods across Cityscapes, PASCAL-Person-Part, and PASCAL VOC 2012?

Key findings

• A random search over the Dense Prediction Cell space can identify architectures that outperform human-designed baselines on three dense prediction tasks.
• The best DPC architecture achieved 82.7% mIOU on Cityscapes test, surpassing the prior state-of-the-art by 0.7 percentage points.
• The same architecture applied to PASCAL-Person-Part and PASCAL VOC 2012 achieved 71.34% and 87.9% mIOU respectively, exceeding prior results.
• The learned DPC required roughly half the parameters and about half the Multiply-Adds (MAdds) compared to previous state-of-the-art systems when using a strong backbone (Xception).
• A designed proxy task using a smaller backbone and cached activations correlates with large-scale performance (Spearman ρ up to about 0.47), enabling efficient architecture search.