[Paper Review] Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots

De Silva, Jamie Roche+1 more|arXiv (Cornell University)|Oct 17, 2017

Target Tracking and Data Fusion in Sensor Networks4 citations

TL;DR

This paper proposes a robust sensor fusion framework that spatially aligns and resolution-matches LiDAR and wide-angle monocular camera data using a geometric model and Gaussian Process regression to improve free space detection in autonomous mobile robots. The method achieves significant performance gains in uncertainty-aware perception by enabling precise, uncertainty-quantified data fusion across heterogeneous sensors with differing spatial and temporal resolutions.

ABSTRACT

Autonomous robots that assist humans in day to day living tasks are becoming increasingly popular. Autonomous mobile robots operate by sensing and perceiving their surrounding environment to make accurate driving decisions. A combination of several different sensors such as LiDAR, radar, ultrasound sensors and cameras are utilized to sense the surrounding environment of autonomous vehicles. These heterogeneous sensors simultaneously capture various physical attributes of the environment. Such multimodality and redundancy of sensing need to be positively utilized for reliable and consistent perception of the environment through sensor data fusion. However, these multimodal sensor data streams are different from each other in many ways, such as temporal and spatial resolution, data format, and geometric alignment. For the subsequent perception algorithms to utilize the diversity offered by multimodal sensing, the data streams need to be spatially, geometrically and temporally aligned with each other. In this paper, we address the problem of fusing the outputs of a Light Detection and Ranging (LiDAR) scanner and a wide-angle monocular image sensor for free space detection. The outputs of LiDAR scanner and the image sensor are of different spatial resolutions and need to be aligned with each other. A geometrical model is used to spatially align the two sensor outputs, followed by a Gaussian Process (GP) regression-based resolution matching algorithm to interpolate the missing data with quantifiable uncertainty. The results indicate that the proposed sensor data fusion framework significantly aids the subsequent perception steps, as illustrated by the performance improvement of a uncertainty aware free space detection algorithm

Motivation & Objective

• To address the challenge of fusing heterogeneous sensor data from LiDAR and wide-angle cameras in autonomous mobile robots.
• To achieve accurate spatial, geometric, and temporal alignment between low-resolution images and high-resolution LiDAR scans.
• To resolve discrepancies in spatial resolution through uncertainty-quantified interpolation using Gaussian Process regression.
• To enhance the performance of downstream perception tasks, particularly uncertainty-aware free space detection.
• To enable reliable, multimodal environmental perception by leveraging complementary strengths of LiDAR and monocular vision.

Proposed method

• A geometric model is used to establish spatial and angular alignment between LiDAR point clouds and wide-angle camera images.
• Gaussian Process regression is applied to interpolate missing data in the image domain to match the resolution of the LiDAR output.
• The GP regression provides uncertainty estimates for interpolated values, enabling robustness in downstream perception.
• The fused data stream combines high-precision depth information from LiDAR with rich texture and semantic cues from the camera.
• The framework ensures temporal consistency by synchronizing sensor data streams based on timestamp alignment.
• The final fused output is used as input to an uncertainty-aware free space detection algorithm.

Experimental results

Research questions

• RQ1 How can LiDAR and wide-angle camera data be effectively spatially and geometrically aligned despite differing resolutions and fields of view?
• RQ2 What is the most effective way to interpolate low-resolution image data to match the resolution of high-precision LiDAR scans while quantifying uncertainty?
• RQ3 To what extent does the proposed fusion framework improve the accuracy and robustness of free space detection in autonomous mobile robots?
• RQ4 How does uncertainty quantification in the fusion process enhance the reliability of perception systems?
• RQ5 Can the fusion framework maintain performance under real-world sensor noise and misalignment?

Key findings

• The proposed fusion framework significantly improves the performance of uncertainty-aware free space detection algorithms.
• Gaussian Process regression enables effective resolution matching with quantifiable uncertainty, enhancing reliability in perception tasks.
• Spatial and geometric alignment via the proposed model reduces misregistration errors between LiDAR and camera data.
• The method demonstrates robustness to sensor noise and variations in environmental conditions due to uncertainty-aware interpolation.
• The fusion approach enables better detection of free space, especially in complex or cluttered environments.
• The results show that multimodal fusion with uncertainty quantification leads to more consistent and accurate perception outputs.