[Paper Review] Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture

David Eigen, Rob Fergus|arXiv (Cornell University)|Nov 18, 2014

Advanced Vision and Imaging37 references127 citations

TL;DR

This paper proposes a unified multi-scale convolutional neural network that predicts depth, surface normals, and semantic labels from a single RGB image using a shared architecture. By progressively refining predictions across three scales—coarse, medium, and fine—the model achieves state-of-the-art performance on all three tasks without relying on superpixels or low-level segmentation, enabling real-time inference at ~30Hz.

ABSTRACT

In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling. We use a multiscale convolutional network that is able to adapt easily to each task using only small modifications, regressing from the input image to the output map directly. Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation. We achieve state-of-the-art performance on benchmarks for all three tasks.

Motivation & Objective

• To develop a single, shared deep learning architecture capable of predicting multiple scene understanding outputs: depth, surface normals, and semantic labels.
• To eliminate reliance on handcrafted components like superpixels or low-level segmentation by leveraging hierarchical multi-scale feature learning.
• To improve performance across diverse tasks using a unified network that progressively refines predictions at increasing resolutions.
• To enable real-time inference (≈30Hz) for all three outputs, facilitating deployment in robotics and augmented reality.
• To demonstrate that a single architecture can outperform task-specific models on depth, normals, and semantic segmentation benchmarks.

Proposed method

• The model uses a multi-scale convolutional network with three stacked scales: coarse (low-resolution global view), medium, and fine (high-resolution refinement).
• At each scale, features are extracted and predictions are refined using deconvolutional layers to upsample and refine the output map.
• The coarse scale provides a global context crucial for depth and normals prediction, while finer scales capture local details.
• The network is trained end-to-end with task-specific loss functions: L1 loss for depth, angular loss for normals, and cross-entropy for semantic segmentation.
• Feature maps from earlier scales are concatenated with later ones to allow local refinements to be informed by global context.
• The architecture is initialized with ImageNet pre-trained weights for the coarsest scale, improving convergence and performance.

Experimental results

Research questions

• RQ1 Can a single multi-scale convolutional network architecture achieve state-of-the-art performance across three distinct computer vision tasks: depth prediction, surface normal estimation, and semantic labeling?
• RQ2 How does the inclusion of a coarse global receptive field (scale 1) impact performance on depth and normals prediction compared to local refinement only?
• RQ3 To what extent can predicted depth and normals from the same network improve semantic segmentation performance when used as auxiliary inputs?
• RQ4 Does the proposed architecture outperform task-specific models that rely on superpixels, CRFs, or complex post-processing?
• RQ5 Can the model generate all three outputs in real time (~30Hz) without sacrificing accuracy?

Key findings

• The multi-scale architecture achieves state-of-the-art performance on all three tasks: depth prediction (0.198 average relative error on NYU Depth v2), surface normals (75.3% mean angle accuracy), and 13-class semantic segmentation (64.0% pixel accuracy).
• The coarsest scale (Scale 1) contributes the most to depth and normals prediction, highlighting the importance of global context, while the medium scale (Scale 2) is most influential for semantic segmentation.
• Using predicted depth and normals as input to the semantic segmentation head improves performance over RGB-only input when only Scale 2 is used, but provides minimal gain when both scales are used—indicating the network can learn these cues independently.
• The model achieves 64.0% pixel accuracy on 13-class semantic segmentation with RGB input alone, surpassing prior methods that use superpixels or CRFs.
• The network runs at approximately 30Hz in inference, enabling real-time deployment across all three tasks.
• Even when randomly initialized, the coarse scale alone achieves 54.5% accuracy on 13-class segmentation, confirming its critical role in global scene understanding.