[Paper Review] SeasonDepth: Cross-Season Monocular Depth Prediction Dataset and Benchmark under Multiple Environments

Hanjiang Hu, Baoquan Yang+7 more|arXiv (Cornell University)|Nov 9, 2020

Advanced Vision and Imaging72 references4 citations

TL;DR

This paper introduces SeasonDepth, the first cross-season monocular depth prediction dataset and benchmark collected across multiple real-world environments (e.g., varying lighting, foliage, and weather) using Structure from Motion (SfM) and Multi-View Stereo (MVS) pipelines. It evaluates state-of-the-art supervised and self-supervised depth models, revealing that current methods struggle with generalization across environmental changes, and demonstrates that fine-tuning on SeasonDepth improves robustness over KITTI or Cityscapes pretraining.

ABSTRACT

Different environments pose a great challenge to the outdoor robust visual perception for long-term autonomous driving, and the generalization of learning-based algorithms on different environments is still an open problem. Although monocular depth prediction has been well studied recently, few works focus on the robustness of learning-based depth prediction across different environments, e.g. changing illumination and seasons, owing to the lack of such a multi-environment real-world dataset and benchmark. To this end, the first cross-season monocular depth prediction dataset and benchmark, SeasonDepth, is introduced to benchmark the depth estimation performance under different environments. We investigate several state-of-the-art representative open-source supervised and self-supervised depth prediction methods using newly-formulated metrics. Through extensive experimental evaluation on the proposed dataset and cross-dataset evaluation with current autonomous driving datasets, the performance and robustness against the influence of multiple environments are analyzed qualitatively and quantitatively. We show that long-term monocular depth prediction is still challenging and believe our work can boost further research on the long-term robustness and generalization for outdoor visual perception. The dataset is available on https://seasondepth.github.io, and the benchmark toolkit is available on https://github.com/ SeasonDepth/SeasonDepth.

Motivation & Objective

• To address the lack of real-world datasets that evaluate monocular depth prediction across diverse environmental conditions such as seasonal changes, lighting, and weather.
• To benchmark the generalization and robustness of state-of-the-art supervised and self-supervised depth prediction models under multiple environmental variations.
• To provide a publicly available dataset and benchmark toolkit to enable systematic evaluation and future research on long-term robustness in outdoor visual perception.
• To analyze how environmental shifts affect depth prediction performance quantitatively and qualitatively, identifying failure modes and promising directions.

Proposed method

• Constructed the SeasonDepth dataset using the CMU Visual Localization dataset, applying SfM and MVS pipelines with RANSAC and manual post-processing to generate scaleless dense depth maps.
• Collected RGB images and corresponding depth maps along the same urban routes under multiple environmental conditions: cloudy, overcast, low sun, and varying foliage.
• Proposed new evaluation metrics including mean, variance, and relative range to assess performance consistency across environments.
• Benchmarked representative open-source models from KITTI and Cityscapes leaderboards, including supervised, stereo-based, video-based, and domain adaptation methods.
• Performed cross-dataset evaluations by fine-tuning models on SeasonDepth and testing on KITTI, and vice versa, to assess domain generalization.
• Applied mean and variance alignment to scaleless depth predictions for fair comparison with stereo-based ground truth.

Experimental results

Research questions

• RQ1 How do state-of-the-art monocular depth prediction models perform across different environmental conditions such as seasonal changes and varying illumination?
• RQ2 What is the impact of domain shift when fine-tuning models trained on KITTI or Cityscapes on the SeasonDepth dataset?
• RQ3 Which depth prediction methods show better robustness and consistency in terms of mean performance and variance across diverse environments?
• RQ4 To what extent does training on SeasonDepth improve generalization to other datasets like KITTI compared to training on Cityscapes?

Key findings

• Most well-tuned depth prediction models exhibit poor generalization, showing high variance and inconsistent performance across different environmental conditions such as low sun or overcast skies.
• Fine-tuning on SeasonDepth leads to better performance on KITTI than fine-tuning on Cityscapes, indicating that SeasonDepth captures more diverse and challenging environmental variations.
• Models fine-tuned on SeasonDepth show significantly lower performance fluctuation (measured by absErrorRel and sqErrorRel) compared to those fine-tuned on Cityscapes, suggesting improved robustness.
• The performance of KITTI-pretrained models degrades substantially on SeasonDepth, highlighting a significant domain shift and the limitations of current models in handling environmental changes.
• Despite being scaleless, the SfM/MVS-based depth maps in SeasonDepth are sufficiently accurate for training and evaluation, as validated by cross-dataset performance comparisons.