[Paper Review] IM2HEIGHT: Height Estimation from Single Monocular Imagery via Fully Residual Convolutional-Deconvolutional Network

Lichao Mou, Xiao Xiang Zhu|arXiv (Cornell University)|Feb 28, 2018

Remote Sensing and LiDAR Applications33 references101 citations

TL;DR

The paper proposes a fully residual convolutional-deconvolutional network to predict height maps from a single monocular remote sensing image, using DSM as ground truth, with skip connections to preserve edges; evaluated on Berlin data showing quantitative improvements over baselines and an application to building instance segmentation.

ABSTRACT

In this paper we tackle a very novel problem, namely height estimation from a single monocular remote sensing image, which is inherently ambiguous, and a technically ill-posed problem, with a large source of uncertainty coming from the overall scale. We propose a fully convolutional-deconvolutional network architecture being trained end-to-end, encompassing residual learning, to model the ambiguous mapping between monocular remote sensing images and height maps. Specifically, it is composed of two parts, i.e., convolutional sub-network and deconvolutional sub-network. The former corresponds to feature extractor that transforms the input remote sensing image to high-level multidimensional feature representation, whereas the latter plays the role of a height generator that produces height map from the feature extracted from the convolutional sub-network. Moreover, to preserve fine edge details of estimated height maps, we introduce a skip connection to the network, which is able to shuttle low-level visual information, e.g., object boundaries and edges, directly across the network. To demonstrate the usefulness of single-view height prediction, we show a practical example of instance segmentation of buildings using estimated height map. This paper, for the first time in the remote sensing community, attempts to estimate height from monocular vision. The proposed network is validated using a large-scale high resolution aerial image data set covered an area of Berlin. Both visual and quantitative analysis of the experimental results demonstrate the effectiveness of our approach.

Motivation & Objective

• Address height estimation from a single monocular remote sensing image, an intrinsically ill-posed problem.
• Develop a fully end-to-end residual conv-deconv network to map RGB images to height maps using DSM as ground truth.
• Preserve edge details in height maps via skip connections and evaluate practical usefulness (e.g., building instance segmentation).

Proposed method

• Two-part architecture: a convolutional sub-network for feature extraction and a deconvolutional sub-network as a height generator.
• Adopt residual blocks to enable residual learning and ease optimization.
• Incorporate a skip connection between the first block and the next-to-last block to shuttle low-level edge information across the network.
• Use unpooling with max-pooling indices in the deconvolutional path to better preserve spatial details.
• Train end-to-end with MAE loss on RGB inputs to predict high-resolution height maps; data augmentation and very small batch size are used.
• Compared against Eigen-Net and plain conv-deconv baselines to assess edge preservation and height map quality.

Experimental results

Research questions

• RQ1 Can a single monocular remote sensing image be mapped to a physically plausible height map using an end-to-end deep network?
• RQ2 Does residual learning with a convolutional-deconvolutional architecture improve height map accuracy and edge preservation compared to non-residual or plain architectures?
• RQ3 How does the proposed method compare to existing single-image height estimation approaches (e.g., Eigen-Net) on a high-resolution Berlin dataset?
• RQ4 Is the predicted height map useful for downstream tasks such as instance segmentation of buildings?

Key findings

• On the Berlin test scene, the res. conv-deconv net with skip connection achieves MSE 7.8e-04, MAE 0.017, and SSIM 0.9366, outperforming baselines.
• The plain conv-deconv network performs poorly and fails to learn realistic height maps, while the residual with skip connection yields substantial edge-preserving improvements.
• Compared to Eigen-Net (which yields blurred, lower-resolution predictions), the proposed method provides higher quality, edge-preserving height maps without post-processing.
• Qualitative results show sharper object boundaries and better structural detail in height maps across diverse land-use scenes.
• An application demonstration shows buildings can be segmented from predicted height maps by thresholding and vegetation filtering, without pixel-level supervision.
• Some failure cases persist for tall buildings, indicating remaining challenges for high-rise height estimation.