[論文レビュー] DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data

Wei Yin, Xinlong Wang+6 more|arXiv (Cornell University)|Feb 3, 2020

Advanced Vision and Imaging参考文献 30被引用数 34

ひとこと要約

DiverseDepth は、大規模で多様なデータセットからアファイン不変の深度を学習し、マルチカリキュラム学習戦略を用いて、幾何学的なシーン構造を保持しつつ、モノキュラ深度推定のゼロショット一般化性能を高く実現します。

ABSTRACT

We present a method for depth estimation with monocular images, which can predict high-quality depth on diverse scenes up to an affine transformation, thus preserving accurate shapes of a scene. Previous methods that predict metric depth often work well only for a specific scene. In contrast, learning relative depth (information of being closer or further) can enjoy better generalization, with the price of failing to recover the accurate geometric shape of the scene. In this work, we propose a dataset and methods to tackle this dilemma, aiming to predict accurate depth up to an affine transformation with good generalization to diverse scenes. First we construct a large-scale and diverse dataset, termed Diverse Scene Depth dataset (DiverseDepth), which has a broad range of scenes and foreground contents. Compared with previous learning objectives, i.e., learning metric depth or relative depth, we propose to learn the affine-invariant depth using our diverse dataset to ensure both generalization and high-quality geometric shapes of scenes. Furthermore, in order to train the model on the complex dataset effectively, we propose a multi-curriculum learning method. Experiments show that our method outperforms previous methods on 8 datasets by a large margin with the zero-shot test setting, demonstrating the excellent generalization capacity of the learned model to diverse scenes. The reconstructed point clouds with the predicted depth show that our method can recover high-quality 3D shapes. Code and dataset are available at: https://tinyurl.com/DiverseDepth

研究の動機と目的

• 多様なシーン全体で一般化する深度推定を動機づけ、正確な3D幾何を保持する。
• 剛体/非剛体を含む、室内外シーンを跨ぐ大規模で多様なRGB-Dデータセット（DiverseDepth）を構築する。
• スケール/平行移動を深度から分離し、アファイン不変な深度推定を提案して、より良い一般化を実現する。
• 複雑で多様なデータに効果的に適用するマルチカリキュラム学習方式を開発する。

提案手法

• DiverseDepth データセットを以下の3部に導入する: Part-fore (前景), Part-in (室内背景), Part-out (屋外背景)。
• 実カメラ系と仮想カメラ系の間でスケールと平行移動を分離することにより、アファイン不変な深度推定を定式化する。
• 高次幾何制約（仮想法線、表面法線）とスケール-シフト不変損失（SSIL）を組み合わせた損失を用いる。
• 難易度でデータをソートし、3部のデータから易しい～難しいのミニバッチで学習する、マルチカリキュラム学習（MCL）戦略を採用する。
• アファイン不変深度をメトリック深度に再スケーリングした後、8データセットでゼロショットテストを行い、Abs-RelとWHDR指標を用いて評価する。

Figure 1: Qualitative comparison of depth and reconstructed 3D point cloud between our method and that of the recent learning relative depth method of Xian et al. [ 30 ] . The first row is the predicted depth and reconstructed 3D point cloud from the depth of theirs, while the second row is ours. Th

実験結果

リサーチクエスチョン

• RQ1 アファイン不変深度 learned on a diverse dataset generalize to unseen scenes better than metric or relative depth methods?
• RQ2 Does a large, diverse training corpus combined with a structured curriculum improve cross-domain depth prediction quality?
• RQ3 What is the impact of combining VNL/SSIL losses with affine-invariance on 3D shape reconstruction?
• RQ4 How does the proposed method perform on foreground objects (e.g., people) compared to background scenes?

主な発見

• Outperforms prior metric-depth and relative-depth methods on 8 zero-shot datasets, with up to 70% relative improvement mentioned.
• On NYU, achieves performance competitive with methods trained specifically on NYU (11.7% Abs-Rel vs. 12.3% for a competing method).
• The method yields higher-quality 3D reconstructions, preserving scene geometry better than relative-depth baselines.
• Ablation shows multi-curriculum learning substantially boosts generalization over uniform sampling and reversed-curriculum variants.
• Loss analysis shows VNL and SSIL outperform other losses for affine-invariant depth on diverse datasets.

Figure 2: Examples of the DiverseDepth dataset. Purple parts are closer, while red regions are farther.