[Paper Review] Exploiting temporal consistency for real-time video depth estimation

Haokui Zhang, Chunhua Shen+4 more|arXiv (Cornell University)|Aug 10, 2019

Advanced Vision and Imaging45 references13 citations

TL;DR

This paper proposes a real-time video depth estimation framework that leverages temporal consistency using a novel spatial-temporal CLSTM (ST-CLSTM) architecture combining 2D CNNs and convolutional LSTMs to capture both spatial features and inter-frame temporal correlations. It introduces a generative adversarial learning-based temporal consistency loss, achieving state-of-the-art performance with minimal computational overhead and enabling real-time inference at 174.83 fps using parallel data loading.

ABSTRACT

Accuracy of depth estimation from static images has been significantly improved recently, by exploiting hierarchical features from deep convolutional neural networks (CNNs). Compared with static images, vast information exists among video frames and can be exploited to improve the depth estimation performance. In this work, we focus on exploring temporal information from monocular videos for depth estimation. Specifically, we take the advantage of convolutional long short-term memory (CLSTM) and propose a novel spatial-temporal CSLTM (ST-CLSTM) structure. Our ST-CLSTM structure can capture not only the spatial features but also the temporal correlations/consistency among consecutive video frames with negligible increase in computational cost. Additionally, in order to maintain the temporal consistency among the estimated depth frames, we apply the generative adversarial learning scheme and design a temporal consistency loss. The temporal consistency loss is combined with the spatial loss to update the model in an end-to-end fashion. By taking advantage of the temporal information, we build a video depth estimation framework that runs in real-time and generates visually pleasant results. Moreover, our approach is flexible and can be generalized to most existing depth estimation frameworks. Code is available at: https://tinyurl.com/STCLSTM

Motivation & Objective

• Address the challenge of real-time depth estimation in video sequences, particularly for applications like autonomous driving and robotics.
• Leverage temporal information across consecutive video frames to improve depth estimation accuracy beyond static image methods.
• Develop a lightweight, efficient framework that maintains high performance without significant computational cost.
• Ensure temporal consistency in predicted depth maps across video frames to produce visually coherent results.
• Design a generalizable framework compatible with most existing depth estimation architectures.

Proposed method

• Propose a novel ST-CLSTM structure that integrates a 2D CNN for spatial feature extraction with a CLSTM to model temporal dependencies across video frames.
• Use a 3D CNN as a discriminator in a GAN framework to learn a temporal consistency loss that penalizes inconsistent depth predictions across frames.
• Combine the spatial loss (L1 or L2 between predicted and ground-truth depth) with the adversarial temporal loss for end-to-end training.
• Apply parallel data loading (PS-mode) to accelerate inference, achieving real-time performance on standard hardware.
• Train the model end-to-end using both spatial and temporal losses, enabling joint optimization of depth accuracy and temporal smoothness.
• Utilize a shallow ResNet-18 backbone to maintain low inference cost while achieving competitive performance.

Experimental results

Research questions

• RQ1 Can temporal correlations in monocular video sequences be effectively exploited to improve depth estimation beyond static image methods?
• RQ2 How can a deep learning model maintain temporal consistency in predicted depth maps across consecutive video frames?
• RQ3 What is the impact of using a GAN-based adversarial loss on temporal consistency and visual quality of depth estimation?
• RQ4 Can a lightweight ST-CLSTM architecture achieve real-time inference without sacrificing performance?
• RQ5 To what extent can this framework be generalized to existing depth estimation models with minimal architectural changes?

Key findings

• The proposed ST-CLSTM framework achieves state-of-the-art performance on both NYU Depth V2 and KITTI datasets, outperforming many deeper models.
• On the NYU Depth V2 dataset, the method achieves a relative error (Rel) of 0.131, RMS of 0.571, and δ1 of 0.833 using ResNet-18, outperforming methods with deeper backbones.
• On the KITTI dataset, the method achieves a Rel of 0.101, RMS of 4.137, and δ1 of 0.890, surpassing most prior video and static image methods.
• The model runs at 174.83 fps in parallel+serial loading mode, exceeding real-time requirements (30 fps) and demonstrating strong practical feasibility.
• The ablation study confirms that both ST-CLSTM and the GAN-based temporal loss significantly improve temporal consistency and overall performance.
• The framework is highly generalizable and can be integrated into most existing depth estimation architectures with minimal modification.