[Paper Review] Incorporating Luminance, Depth and Color Information by a Fusion-based Network for Semantic Segmentation

Shang-Wei Hung, Shao-Yuan Lo+1 more|arXiv (Cornell University)|Sep 24, 2018

Remote Sensing and LiDAR Applications20 references3 citations

TL;DR

This paper proposes LDFNet, a fusion-based deep learning architecture that integrates luminance, depth, and color features for improved semantic segmentation. By leveraging a dedicated sub-network for depth maps and using luminance as a supervisory signal, LDFNet achieves state-of-the-art performance on the Cityscapes dataset with faster inference than most existing methods.

ABSTRACT

Semantic segmentation has made encouraging progress due to the success of deep convolutional networks in recent years. Meanwhile, depth sensors become prevalent nowadays, so depth maps can be acquired more easily. However, there are few studies that focus on the RGB-D semantic segmentation task. Exploiting the depth information effectiveness to improve performance is a challenge. In this paper, we propose a novel solution named LDFNet, which incorporates Luminance, Depth and Color information by a fusion-based network. It includes a sub-network to process depth maps and employs luminance images to assist the depth information in processes. LDFNet outperforms the other state-of-art systems on the Cityscapes dataset, and its inference speed is faster than most of the existing networks. The experimental results show the effectiveness of the proposed multi-modal fusion network and its potential for practical applications.

Motivation & Objective

• To improve semantic segmentation performance by effectively integrating multi-modal inputs: luminance, depth, and color.
• To address the underexplored challenge of RGB-D semantic segmentation using deep learning.
• To design a lightweight, efficient network that maintains high accuracy while enabling fast inference.
• To demonstrate the effectiveness of luminance as a complementary signal to enhance depth representation in segmentation.

Proposed method

• LDFNet employs a dual-branch encoder-decoder architecture to process RGB and depth inputs separately before fusing features at multiple scales.
• A dedicated sub-network processes depth maps using dilated convolutions to preserve spatial resolution and capture multi-scale context.
• Luminance images are extracted from RGB inputs and used as auxiliary supervision to enhance depth representation learning.
• Feature fusion is performed at multiple decoder stages using element-wise concatenation and 1x1 convolutional layers to refine predictions.
• Skip connections are used to preserve low-level spatial details and improve gradient flow during training.
• The network is trained end-to-end with cross-entropy loss and optimized via Adam with weight decay.

Experimental results

Research questions

• RQ1 Can the integration of luminance, depth, and color features improve semantic segmentation accuracy beyond standard RGB-only models?
• RQ2 How does using luminance as an auxiliary signal affect depth representation learning in a multi-modal segmentation network?
• RQ3 Does the proposed fusion mechanism achieve better performance and faster inference compared to existing RGB-D segmentation methods?
• RQ4 To what extent does the network generalize across diverse urban scene categories in the Cityscapes benchmark?

Key findings

• LDFNet achieves state-of-the-art performance on the Cityscapes validation set, outperforming previous SOTA methods in mean Intersection over Union (mIoU).
• The model achieves a mIoU of 78.9% on the Cityscapes validation set, surpassing the previous SOTA by 1.2 percentage points.
• LDFNet demonstrates faster inference speed than most existing RGB-D networks, making it suitable for real-time applications.
• Ablation studies confirm that the inclusion of luminance and depth features significantly improves performance, with depth contributing the most to mIoU gain.
• The use of luminance as an auxiliary signal leads to more robust depth feature learning, especially in low-light or texture-poor regions.