[Paper Review] PackNet-SfM: 3D Packing for Self-Supervised Monocular Depth Estimation.

Vitor Guizilini, Rareş Ambruş+2 more|arXiv (Cornell University)|May 6, 2019

Advanced Vision and Imaging26 citations

TL;DR

PackNet-SfM proposes a novel 3D packing and unpacking architecture for self-supervised monocular depth estimation using monocular video, introducing a velocity supervision loss to enable metric depth prediction without test-time scaling. It achieves state-of-the-art performance on KITTI, outperforming all monocular-video-trained methods and rivaling stereo-trained models.

ABSTRACT

Densely estimating the depth of a scene from a single image is an ill-posed inverse problem that is seeing exciting progress with self-supervision from strong geometric cues, in particular from training using stereo imagery. In this work, we investigate the more challenging structure-from-motion (SfM) setting, learning purely from monocular videos. We propose PackNet - a novel deep architecture that leverages new 3D packing and unpacking blocks to effectively capture fine details in monocular depth map predictions. Additionally, we propose a novel velocity supervision loss that allows our model to predict metrically accurate depths, thus alleviating the need for test-time ground-truth scaling. We show that our proposed scale-aware architecture achieves state-of-the-art results on the KITTI benchmark, significantly improving upon any approach trained on monocular video, and even achieves competitive performance to stereo-trained methods.

Motivation & Objective

• To address the challenge of self-supervised monocular depth estimation from monocular video, which lacks the geometric constraints of stereo supervision.
• To improve fine-grained depth detail capture in single-image depth prediction using a novel 3D packing and unpacking mechanism.
• To enable metrically accurate depth predictions without requiring test-time ground-truth scaling, a major limitation in self-supervised monocular methods.
• To close the performance gap between monocular-video-trained and stereo-trained depth estimation models.

Proposed method

• Introduces 3D packing and unpacking blocks that process feature maps across spatial, channel, and depth dimensions to enhance feature representation and detail recovery.
• Employs a novel velocity supervision loss based on monocular video motion consistency to predict metric-scale depths without test-time scaling.
• Leverages structure-from-motion (SfM) cues from monocular video sequences to provide geometric supervision during training.
• Uses a multi-scale encoder-decoder architecture with 3D packing to model long-range dependencies and preserve fine details in depth maps.
• Applies inverse depth supervision with a differentiable warping layer to enforce photometric consistency across video frames.
• Optimizes the network using a combination of depth supervision, photometric consistency, and velocity-based metric calibration.

Experimental results

Research questions

• RQ1 Can a self-supervised monocular depth model trained solely on video sequences achieve state-of-the-art performance without stereo supervision?
• RQ2 How effective is 3D packing and unpacking in capturing fine details in depth maps compared to standard 2D feature processing?
• RQ3 Can velocity-based supervision enable metric depth prediction without requiring test-time scaling or depth supervision?
• RQ4 To what extent does the proposed method close the performance gap between monocular-video and stereo-trained depth estimation models?

Key findings

• PackNet-SfM achieves state-of-the-art depth estimation performance on the KITTI benchmark among models trained exclusively on monocular video.
• The model significantly outperforms prior monocular-video self-supervised methods in terms of absolute relative error (Abs Rel), squared relative error (Sq Rel), and root mean square error (RMSE).
• The velocity supervision loss enables metric depth prediction without test-time scaling, eliminating a common source of error in self-supervised methods.
• The 3D packing and unpacking blocks improve feature representation and preserve fine details, especially in complex scenes with depth discontinuities.
• The model achieves competitive performance compared to stereo-trained methods, demonstrating the potential of monocular video to deliver high-quality depth estimation.