[Paper Review] LiDAR Sensor modeling and Data augmentation with GANs for Autonomous driving

Ahmad El Sallab, Ibrahim Sobh+2 more|arXiv (Cornell University)|May 17, 2019

Advanced Neural Network Applications16 references33 citations

TL;DR

The paper formulates LiDAR sensor modeling as unpaired image-to-image translation using CycleGANs to map simulated CARLA LiDAR to real KITTI LiDAR (sim2real) and to perform real-to-real translation for LiDAR representations, enabling data augmentation and realism improvements.

ABSTRACT

In the autonomous driving domain, data collection and annotation from real vehicles are expensive and sometimes unsafe. Simulators are often used for data augmentation, which requires realistic sensor models that are hard to formulate and model in closed forms. Instead, sensors models can be learned from real data. The main challenge is the absence of paired data set, which makes traditional supervised learning techniques not suitable. In this work, we formulate the problem as image translation from unpaired data and employ CycleGANs to solve the sensor modeling problem for LiDAR, to produce realistic LiDAR from simulated LiDAR (sim2real). Further, we generate high-resolution, realistic LiDAR from lower resolution one (real2real). The LiDAR 3D point cloud is processed in Bird-eye View and Polar 2D representations. The experimental results show a high potential of the proposed approach.

Motivation & Objective

• Motivate the need for realistic LiDAR sensor models to augment data for autonomous driving without risky real-world collection.
• Propose an unsupervised learning approach to model LiDAR sensing using unpaired data from simulation and real data.
• Investigate two LiDAR representations (Bird-eye View BEV and Polar Grid Map PGM) for translation between domains.
• Demonstrate sim2real and real2real translations to produce realistic LiDAR data from simulations and upscale resolution.
• Discuss evaluation strategies for generated LiDAR data and outline future improvements.

Proposed method

• Formulate LiDAR sensor modeling as an unpaired image-to-image translation problem using CycleGANs.
• Translate between two domains: simulated LiDAR from CARLA and real LiDAR from KITTI, using BEV and PGM projections.
• Define GAN losses for both directions (G: sim2real, F: real2sim) along with cycle-consistency losses.
• Apply two 2D LiDAR representations (BEV and 2D PGM) to preserve 3D information in a 2D framework.
• Optionally extend CycleGAN with task-specific and extrinsic losses to preserve object-level content.
• Train with datasets: CARLA as X and KITTI as Y, exploring 2000-frame subsets for experiments.

Experimental results

Research questions

• RQ1 Can CycleGANs learn realistic mappings between simulated and real LiDAR data without paired samples?
• RQ2 Do BEV and PGM representations preserve enough LiDAR information for effective sim2real translation?
• RQ3 Is it possible to translate to higher-resolution LiDAR from lower-resolution inputs using unsupervised image-to-image translation?
• RQ4 Can extrinsic evaluation methods (e.g., object-detection-oriented metrics) quantify the quality of generated LiDAR data?
• RQ5 What are viable strategies to evaluate and augment LiDAR data for downstream perception tasks?

Key findings

• CycleGAN-based translation can generate KITTI-like BEV from CARLA inputs, preserving the general KITTI characteristics.
• Real2real translation shows the model can transfer content when moving from sparser to denser LiDAR channel configurations.
• PGM-based translations demonstrate that 3D LiDAR information can be encoded and translated in a dense 2D representation.
• The study discusses multiple evaluation avenues, including reconstruction quality, annotation transfer, and extrinsic task-based assessments.
• The authors propose a framework to augment data and improve realism, with potential future extensions using task-specific loss terms and extrinsic evaluators.