[Paper Review] Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes

Yuanduo Hong, Huihui Pan+2 more|arXiv (Cornell University)|Jan 15, 2021

Advanced Neural Network Applications60 references254 citations

TL;DR

DDRNet introduces dual-resolution deep branches with bilateral fusion and a Deep Aggregation Pyramid Pooling Module to achieve high-accuracy real-time semantic segmentation of road scenes, setting new speed-accuracy trade-offs on Cityscapes and CamVid.

ABSTRACT

Semantic segmentation is a key technology for autonomous vehicles to understand the surrounding scenes. The appealing performances of contemporary models usually come at the expense of heavy computations and lengthy inference time, which is intolerable for self-driving. Using light-weight architectures (encoder-decoder or two-pathway) or reasoning on low-resolution images, recent methods realize very fast scene parsing, even running at more than 100 FPS on a single 1080Ti GPU. However, there is still a significant gap in performance between these real-time methods and the models based on dilation backbones. To tackle this problem, we proposed a family of efficient backbones specially designed for real-time semantic segmentation. The proposed deep dual-resolution networks (DDRNets) are composed of two deep branches between which multiple bilateral fusions are performed. Additionally, we design a new contextual information extractor named Deep Aggregation Pyramid Pooling Module (DAPPM) to enlarge effective receptive fields and fuse multi-scale context based on low-resolution feature maps. Our method achieves a new state-of-the-art trade-off between accuracy and speed on both Cityscapes and CamVid dataset. In particular, on a single 2080Ti GPU, DDRNet-23-slim yields 77.4% mIoU at 102 FPS on Cityscapes test set and 74.7% mIoU at 230 FPS on CamVid test set. With widely used test augmentation, our method is superior to most state-of-the-art models and requires much less computation. Codes and trained models are available online.

Motivation & Objective

• Motivate real-time semantic segmentation for autonomous driving with high accuracy.
• Develop efficient backbones that balance high-resolution representations and semantic context.
• Isolate how dual-resolution fusion levels impact speed and accuracy.
• Introduce a lightweight, context-rich module (DAPPM) that enhances low-resolution features without heavy cost.

Proposed method

• Design the DDRNet family with two deep branches at different resolutions.
• Implement bilateral fusion between high- and low-resolution branches at multiple stages.
• Propose the Deep Aggregation Pyramid Pooling Module (DAPPM) to capture multi-scale context on low-resolution maps.
• Use a simple segmentation head with adjustable output channels to control compute.
• Apply deep supervision during training to stabilize optimization.
• Pre-train on ImageNet before fine-tuning on semantic segmentation datasets.

Experimental results

Research questions

• RQ1 Can dual-resolution networks with bilateral fusion achieve higher real-time segmentation accuracy than single-resolution or dilated-backbone methods?
• RQ2 Does the Deep Aggregation Pyramid Pooling Module provide richer multi-scale context on low-resolution features with negligible speed impact?
• RQ3 What is the optimal trade-off between network depth/width and inference speed for road-scene segmentation?
• RQ4 How does DDRNet perform across Cityscapes, CamVid, and COCOStuff compared to state-of-the-art real-time models?

Key findings

• DDRNet-23-slim achieves 77.4% mIoU at 102 FPS on Cityscapes test set.
• DDRNet-23 achieves 79.4% mIoU on Cityscapes test at 37 FPS.
• DDRNet-39 reaches 80.4% mIoU on Cityscapes test at 22 FPS.
• On CamVid, DDRNet-23-slim attains 74.7% mIoU at 230 FPS without Cityscapes pre-training.
• On COCOStuff, DDRNet-23 and DDRNet-39 achieve 32.1 and 34.8 mIoU respectively at high speed.
• With Cityscapes pre-training and TensorRT acceleration, DDRNet-23 achieves 80.6% mIoU at 94 FPS on Cityscapes.