[Paper Review] Rethinking Skip Connections in Encoder-decoder Networks for Monocular Depth Estimation

Zhitong Lai, Haichao Sun+4 more|arXiv (Cornell University)|Aug 29, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a Full Skip Connection Network (FSCN) for monocular depth estimation that enhances feature propagation by connecting all encoder and decoder feature levels, not just same-resolution features. It introduces an Adaptive Concatenation Module (ACM) with learnable weights and channel attention to improve feature fusion, achieving state-of-the-art performance on KITTI and NYU Depth V2 datasets with lower error and higher accuracy metrics.

ABSTRACT

Skip connections are fundamental units in encoder-decoder networks, which are able to improve the feature propagtion of the neural networks. However, most methods with skip connections just connected features with the same resolution in the encoder and the decoder, which ignored the information loss in the encoder with the layers going deeper. To leverage the information loss of the features in shallower layers of the encoder, we propose a full skip connection network (FSCN) for monocular depth estimation task. In addition, to fuse features within skip connections more closely, we present an adaptive concatenation module (ACM). Further more, we conduct extensive experiments on the ourdoor and indoor datasets (i.e., the KITTI dataste and the NYU Depth V2 dataset) for FSCN and FSCN gets the state-of-the-art results.

Motivation & Objective

• To address information loss in deep encoder-decoder networks by rethinking skip connections beyond same-resolution feature fusion.
• To improve feature representation in monocular depth estimation by leveraging multi-level features from both encoder and decoder.
• To design a more effective feature fusion mechanism than standard concatenation for skip connections.
• To achieve state-of-the-art performance on standard monocular depth estimation benchmarks (KITTI and NYU Depth V2).

Proposed method

• Proposes a Full Skip Connection Network (FSCN) that establishes dense skip connections between all spatial levels of the encoder and decoder, rather than only matching same-resolution features.
• Introduces an Adaptive Concatenation Module (ACM) that learns channel-wise weights and applies Squeeze-Excitation (SE) attention to dynamically balance feature fusion.
• Uses learnable concatenation weights to adaptively scale features from different levels before concatenation, improving fusion quality.
• Employs a U-Net-like encoder-decoder architecture with residual blocks and skip connections for depth prediction.
• Trains the network end-to-end using supervised loss functions on RGB-D datasets, with depth range normalization.
• Validates the method on KITTI and NYU Depth V2 datasets using standard metrics like abs_rel, log10, rms, and δ<1.25.

Experimental results

Research questions

• RQ1 Can dense skip connections between all encoder and decoder feature levels reduce information loss and improve depth estimation accuracy?
• RQ2 How does adaptive feature fusion via learnable weights and attention mechanisms compare to standard concatenation in skip connections?
• RQ3 Does the proposed FSCN architecture outperform existing encoder-decoder networks with standard skip connections on monocular depth estimation benchmarks?
• RQ4 What is the relative contribution of the concatenation weights and the Squeeze-Excitation module in the adaptive concatenation module?

Key findings

• FSCN achieves state-of-the-art performance on the KITTI dataset, reducing abs_rel from 0.112 (same-skip) to 0.111 and improving δ<1.25 from 0.878 to 0.884.
• On the NYU Depth V2 dataset, FSCN achieves the best results with abs_rel = 0.111, log10 = 0.047, and δ<1.25 = 0.884, outperforming prior methods.
• Ablation studies show that removing either the concatenation weights or the Squeeze-Excitation module significantly degrades performance, confirming their importance.
• The ACM with both components achieves the best results, with the Squeeze-Excitation module contributing more than the learnable weights alone.
• The full skip connection mechanism consistently improves performance over standard skip connections, even when the baseline without skip connections performs surprisingly well.
• The proposed method generalizes well across datasets, achieving SOTA on both outdoor (KITTI) and indoor (NYU Depth V2) benchmarks.