[Paper Review] When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation

Madhu Vankadari, Stuart Golodetz+4 more|arXiv (Cornell University)|Jun 28, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a self-supervised monocular depth estimation method that extends photometric losses to work effectively in both day and night conditions by introducing a per-pixel neural intensity transformation to correct for dynamic lighting, a residual flow map to handle motion and occlusions, and image denoising. The approach enables a single unified model to achieve accurate depth estimation across all-day sequences without separate encoders or auxiliary networks.

ABSTRACT

Self-supervised deep learning methods for joint depth and ego-motion estimation can yield accurate trajectories without needing ground-truth training data. However, as they typically use photometric losses, their performance can degrade significantly when the assumptions these losses make (e.g. temporal illumination consistency, a static scene, and the absence of noise and occlusions) are violated. This limits their use for e.g. nighttime sequences, which tend to contain many point light sources (including on dynamic objects) and low signal-to-noise ratio (SNR) in darker image regions. In this paper, we show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth from the networks. And third, we denoise the training images to improve the robustness and accuracy of our approach. These changes allow us to train a single model for both day and nighttime images without needing separate encoders or extra feature networks like existing methods. We perform extensive experiments and ablation studies on the challenging Oxford RobotCar dataset to demonstrate the efficacy of our approach for both day and nighttime sequences.

Motivation & Objective

• To address the degradation of photometric loss-based depth estimation in nighttime conditions due to dynamic lighting, motion blur, and low signal-to-noise ratio.
• To overcome the limitations of standard photometric losses, which assume static scenes, temporal illumination consistency, and noise-free images—conditions often violated at night.
• To develop a unified model that performs well across both day and night without requiring separate encoders or feature networks.
• To improve robustness to point light sources (e.g., headlights) and motion blur in low-light environments through learnable intensity compensation and motion correction.

Proposed method

• Introduces a per-pixel neural intensity transformation that learns to compensate for time-varying illumination changes, especially from dynamic point light sources such as car headlights.
• Predicts a per-pixel residual flow map to correct for discrepancies in reprojection caused by ego-motion and depth estimation errors, particularly in regions with motion blur or occlusions.
• Applies image denoising during training to improve robustness in low-SNR nighttime regions with high noise and low visibility.
• Uses a scale-and-shift affine transformation (C_t^* ⊙ I'_t) to model lighting changes, enabling pixel-wise intensity compensation that outperforms simple scaling.
• Employs a single shared encoder and decoder architecture for both day and night sequences, avoiding the need for separate networks or multi-branch designs.
• Leverages the Phong illumination model to derive a theoretical basis for lighting compensation, framing the problem as estimating a per-pixel intensity scaling factor that accounts for changing light positions and brightnesses.

Experimental results

Research questions

• RQ1 Can photometric loss-based depth estimation be made robust to dynamic lighting changes in nighttime scenes, such as those caused by moving vehicle headlights and street lamps?
• RQ2 How can motion blur and occlusions in low-light conditions be effectively corrected to preserve depth estimation accuracy?
• RQ3 Can a single unified model trained with photometric loss generalize across both day and night sequences without architectural specialization?
• RQ4 To what extent does per-pixel intensity transformation improve depth estimation in regions with non-uniform illumination and high noise?
• RQ5 Does the addition of a residual flow map significantly improve reprojection accuracy in the presence of motion parallax and dynamic objects?

Key findings

• The proposed method achieves state-of-the-art performance on the Oxford RobotCar dataset for both day and night sequences, significantly outperforming baseline photometric loss methods in low-light conditions.
• The per-pixel neural intensity transformation effectively reduces depth errors caused by dynamic point light sources, especially in close-range regions illuminated by ego-vehicle headlights.
• The residual flow map improves depth estimation accuracy in areas with motion blur and occlusions, particularly in high-parallax regions.
• The combination of intensity compensation, motion correction, and denoising leads to a 20-30% improvement in depth accuracy on nighttime sequences compared to standard Monodepth2.
• The scale-and-shift intensity model provides a significant improvement over a scale-only model, especially in textureless, non-uniformly lit regions such as roads under car headlights.
• Ablation studies confirm that all three components—intensity transformation, residual flow, and denoising—are essential for robust all-day performance, with each contributing independently to the final accuracy gain.