[Paper Review] DRO: Deep Recurrent Optimizer for Structure-from-Motion.

Xiaodong Gu, Weihao Yuan+4 more|arXiv (Cornell University)|Mar 24, 2021

Advanced Vision and ImagingComputer Science44 references17 citations

TL;DR

This paper proposes DRO, a deep recurrent optimizer for structure-from-motion that iteratively refines depth maps and camera poses using gated recurrent units to minimize a feature-metric cost without gradient computation. It achieves state-of-the-art performance in both supervised and self-supervised settings, outperforming prior methods on KITTI and ScanNet while reducing computational and memory costs.

ABSTRACT

There are increasing interests of studying the structure-from-motion (SfM) problem with machine learning techniques. While earlier methods directly learn a mapping from images to depth maps and camera poses, more recent works enforce multi-view geometry constraints through optimization embedded in the learning framework. This paper presents a novel optimization method based on recurrent neural networks to further exploit the potential of neural networks in SfM. Specifically, our neural optimizer alternately updates the depth and camera poses through iterations to minimize a feature-metric cost, and two gated recurrent units iteratively improve the results by tracing historical information. In this way, our network is a gradient-free zeroth-order optimizer designed for SfM and can be applied to both supervised and self-supervised SfM. Extensive experimental results demonstrate that our method outperforms previous methods and is more efficient in computation and memory consumption than cost-volume-based methods. In particular, our self-supervised method outperforms previous supervised methods on the KITTI and ScanNet datasets. Our source code is available at this https URL.

Motivation & Objective

• To address the limitations of end-to-end learning in structure-from-motion by incorporating geometric constraints through optimization.
• To design a differentiable, recurrent optimization framework that improves depth and camera pose estimation without relying on gradient computation.
• To enable efficient and accurate self-supervised structure-from-motion by leveraging historical optimization states through gated recurrent units.
• To reduce computational and memory overhead compared to cost-volume-based methods while maintaining or improving performance.

Proposed method

• DRO employs two gated recurrent units (GRUs) to iteratively update depth maps and camera poses based on a feature-metric cost function.
• The optimization process alternates between refining depth and pose estimates, using historical states to guide convergence.
• The network acts as a zeroth-order optimizer, avoiding backpropagation through the optimization loop.
• The method is trained end-to-end and can be applied to both supervised and self-supervised SfM setups.
• The feature-metric cost function is minimized through iterative refinement guided by GRU-based memory of past optimization steps.

Experimental results

Research questions

• RQ1 Can a recurrent neural network be effectively used to optimize depth and camera poses in structure-from-motion without gradient computation?
• RQ2 How does a gradient-free, recurrent optimization framework compare to gradient-based or cost-volume-based methods in terms of accuracy and efficiency?
• RQ3 Can a self-supervised DRO method outperform supervised baselines on real-world benchmarks like KITTI and ScanNet?
• RQ4 To what extent does incorporating historical optimization states via GRUs improve convergence and final reconstruction quality?

Key findings

• DRO achieves state-of-the-art performance on both supervised and self-supervised structure-from-motion benchmarks.
• The self-supervised version of DRO outperforms previous supervised methods on the KITTI and ScanNet datasets.
• DRO reduces computational and memory consumption compared to cost-volume-based methods while maintaining or improving accuracy.
• The use of gated recurrent units enables effective tracking of optimization history, improving convergence and final result quality.