[Paper Review] Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling

Sebastian Ramos, Stefan Gehrig+3 more|arXiv (Cornell University)|Dec 20, 2016

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes a novel fusion framework combining deep learning-based semantic segmentation with stereo geometry for detecting small, unexpected road obstacles like lost cargo. By leveraging appearance, context, and 3D geometric cues via a Bayesian probabilistic fusion of semantic and stereo Stixel outputs, the system achieves a 50% relative improvement in detection rate and over 90% detection at 50m, operating in real time at 22 Hz.

ABSTRACT

The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues. To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is used to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests. We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner. We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with relative performance gains of up to 50%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.

Motivation & Objective

• To address the critical yet under-explored challenge of detecting small, unexpected road hazards such as lost cargo.
• To overcome limitations of geometry-only systems by incorporating appearance and contextual cues through deep learning.
• To develop a robust, real-time perception system capable of detecting obstacles as small as 5 cm at distances up to 50 m.
• To reduce false positives while maintaining high detection rates in complex, variable road conditions.
• To create a principled probabilistic fusion framework combining semantic and geometric detection outputs.

Proposed method

• A fully convolutional network (FCN) variant performs pixel-wise semantic labeling into free-space, on-road obstacles, and background to exploit appearance and context.
• A stereo-based system using the FPHT algorithm and statistical hypothesis testing generates 3D obstacle detections via Stixel representation.
• A Bayesian fusion framework combines semantic and geometric detection results using confidence scores and obstacle priors.
• The Stixel representation enables compact, flexible, and robust mid-level obstacle description across both modalities.
• The fusion process uses probabilistic inference to combine evidence from semantic and geometric sources, with confidence thresholds for false positive control.
• The system is optimized for real-time performance, running at 22 Hz on a self-driving platform with hardware acceleration for stereo processing.

Experimental results

Research questions

• RQ1 Can deep learning models effectively detect small, unexpected road obstacles by leveraging appearance and contextual cues, even when objects vary widely in shape and appearance?
• RQ2 How can geometric cues from stereo vision be reliably combined with semantic segmentation to improve detection of low-height obstacles?
• RQ3 What is the optimal way to fuse probabilistic outputs from semantic and geometric detection systems to maximize detection rate while minimizing false positives?
• RQ4 To what extent can a learned background class generalize to unseen obstacles while maintaining robustness to road surface variations and lighting changes?
• RQ5 How does the fusion framework perform under real-world challenges such as curved roads, shadows, and decalibration?

Key findings

• The proposed system achieves a 50% relative improvement in detection rate compared to the state-of-the-art on the Lost and Found dataset.
• The system maintains a detection rate of over 90% for obstacles up to 50 meters away, even for objects as small as 5 cm in height.
• False positive rates are reduced by 13% compared to the state-of-the-art, with only one false positive per 2 kilometers driven in practice.
• The fusion system successfully recovers missed detections from individual modalities, such as when UON misses objects due to background class confusion or when stereo systems fail on small obstacles.
• The system operates in real time at 22 Hz on the self-driving platform, with end-to-end latency of 45 ms for SGM, 35 ms for FPHT, and 40 ms for UON on dedicated hardware.
• The probabilistic fusion framework allows intuitive control of false positives via confidence thresholds, maintaining high detection rates up to a threshold of 0.7.