[Paper Review] Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video

Jianfeng Li, Junqiao Zhao+2 more|arXiv (Cornell University)|May 30, 2021

Advanced Vision and Imaging19 references4 citations

TL;DR

This paper proposes a self-supervised framework for joint estimation of depth, optical flow, and camera pose from video, improving accuracy through a novel attention-based pose network, dynamic object masking, and epipolar geometry constraints. It achieves state-of-the-art performance on KITTI benchmarks, particularly in optical flow and pose estimation, with competitive depth results.

ABSTRACT

Estimating geometric elements such as depth, camera motion, and optical flow from images is an important part of the robot's visual perception. We use a joint self-supervised method to estimate the three geometric elements. Depth network, optical flow network and camera motion network are independent of each other but are jointly optimized during training phase. Compared with independent training, joint training can make full use of the geometric relationship between geometric elements and provide dynamic and static information of the scene. In this paper, we improve the joint self-supervision method from three aspects: network structure, dynamic object segmentation, and geometric constraints. In terms of network structure, we apply the attention mechanism to the camera motion network, which helps to take advantage of the similarity of camera movement between frames. And according to attention mechanism in Transformer, we propose a plug-and-play convolutional attention module. In terms of dynamic object, according to the different influences of dynamic objects in the optical flow self-supervised framework and the depth-pose self-supervised framework, we propose a threshold algorithm to detect dynamic regions, and mask that in the loss function respectively. In terms of geometric constraints, we use traditional methods to estimate the fundamental matrix from the corresponding points to constrain the camera motion network. We demonstrate the effectiveness of our method on the KITTI dataset. Compared with other joint self-supervised methods, our method achieves state-of-the-art performance in the estimation of pose and optical flow, and the depth estimation has also achieved competitive results. Code will be available https://github.com/jianfenglihg/Unsupervised_geometry.

Motivation & Objective

• Address the challenge of estimating depth, optical flow, and camera pose from monocular video without ground-truth supervision.
• Improve generalization and accuracy by leveraging geometric relationships among depth, flow, and pose through joint self-supervision.
• Overcome limitations in existing self-supervised methods by incorporating dynamic object handling, attention mechanisms, and geometric constraints.
• Enhance pose estimation robustness by modeling temporal motion similarity across video frames.
• Reduce error propagation from pose to optical flow by introducing stronger geometric constraints using epipolar geometry.

Proposed method

• Introduce a plug-and-play convolutional attention module in the pose network to exploit temporal similarity in camera motion across frames.
• Propose a threshold-based dynamic region detection method that masks dynamic areas in the loss function for both depth-pose and optical flow self-supervision.
• Apply forward and backward optical flow consistency constraints to improve flow estimation accuracy.
• Incorporate epipolar geometry via the eight-point algorithm to enforce geometric consistency between optical flow and estimated camera pose.
• Use image reconstruction loss through warping to enable self-supervised training without ground-truth labels.
• Jointly optimize depth, optical flow, and pose networks using multi-task loss, with consistency and geometric constraints enhancing supervision.

Experimental results

Research questions

• RQ1 Can attention mechanisms improve the accuracy of self-supervised camera pose estimation by modeling motion continuity across frames?
• RQ2 How can dynamic objects be effectively detected and excluded from supervision to reduce error in depth and flow estimation?
• RQ3 Can epipolar geometry constraints improve the consistency between optical flow and pose estimation in a self-supervised setting?
• RQ4 Does joint self-supervision with geometric and consistency constraints lead to better performance than independent or simpler joint training?
• RQ5 To what extent does the proposed method outperform existing state-of-the-art self-supervised methods on standard benchmarks like KITTI?

Key findings

• The proposed method achieves state-of-the-art performance in optical flow estimation on KITTI 2015, with a 5.66% error rate on the 'all' split and 18.57% on the 'bg' split.
• Pose estimation performance is significantly improved by the attention module, reducing error compared to baseline and prior joint methods.
• The addition of cross-task consistency loss ($L_c^{df}$) and geometric constraint ($L_g$) further improves depth estimation, reducing absolute relative error to 0.138 on the KITTI Eigen split.
• The method outperforms GLNet and other joint self-supervised baselines in optical flow and pose estimation, particularly in handling occlusions and motion errors.
• Qualitative results show accurate depth predictions even in complex scenes, with visualizations demonstrating robustness to texture variations and motion.
• Ablation studies confirm that each component—attention, dynamic masking, and geometric constraints—contributes incrementally to performance gains.