[논문 리뷰] A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving

Xiangyu Yue, BoRui Wu외 3명|ArXiv.org|2018. 03. 31.

Advanced Neural Network Applications참고 문헌 22인용 수 45

한 줄 요약

본 논문은 GTA-V 기반 프레임워크를 통해 자동 보정을 갖춘 주석이 달린 LiDAR 포인트 클라우드를 합성하고, 자율주행을 위한 LiDAR/비전 네트워크의 학습과 테스트를 가능하게 한다; 합성 데이터가 분할 성능을 향상시키고 재학습을 위한 네트워크의 약점을 노출하는 데 도움을 준다.

ABSTRACT

3D LiDAR scanners are playing an increasingly important role in autonomous driving as they can generate depth information of the environment. However, creating large 3D LiDAR point cloud datasets with point-level labels requires a significant amount of manual annotation. This jeopardizes the efficient development of supervised deep learning algorithms which are often data-hungry. We present a framework to rapidly create point clouds with accurate point-level labels from a computer game. The framework supports data collection from both auto-driving scenes and user-configured scenes. Point clouds from auto-driving scenes can be used as training data for deep learning algorithms, while point clouds from user-configured scenes can be used to systematically test the vulnerability of a neural network, and use the falsifying examples to make the neural network more robust through retraining. In addition, the scene images can be captured simultaneously in order for sensor fusion tasks, with a method proposed to do automatic calibration between the point clouds and captured scene images. We show a significant improvement in accuracy (+9%) in point cloud segmentation by augmenting the training dataset with the generated synthesized data. Our experiments also show by testing and retraining the network using point clouds from user-configured scenes, the weakness/blind spots of the neural network can be fixed.

연구 동기 및 목표

• Automate creation of labeled 3D LiDAR point clouds from a virtual world for autonomous driving research.
• Enable automatic calibration between LiDAR point clouds and camera images for sensor fusion.
• Provide a configurable in-game scene to systematically test and improve neural networks via synthetic data and retraining.

제안 방법

• Use Grand Theft Auto V with a virtual LiDAR to collect synchronized point clouds and in-game images.
• Apply automatic calibration to map LiDAR points to image pixels using geometric projections.
• Allow user-configurable scene parameters to generate diverse, controlled test data.
• Evaluate LiDAR point cloud segmentation with IoU, Precision, and Recall on KITTI-derived labels.
• Combine GTA-V synthetic data with KITTI real data to train and validate segmentation models.

실험 결과

연구 질문

• RQ1 Can synthetic LiDAR data from a virtual world improve real-world LiDAR segmentation performance?
• RQ2 How does retraining with synthetic, challenging scenarios affect network robustness and coverage of failure modes?
• RQ3 Can automatic calibration enable effective sensor fusion between LiDAR and images using simulated data?
• RQ4 What are the network weaknesses (blind spots) revealed by systematically sampled in-game scenes?

주요 결과

• Augmenting KITTI with GTA-V synthetic data improves segmentation IoU by about 9% compared to using KITTI alone.
• Purely synthetic GTA-V data performs poorly alone due to distribution gaps, but combining with KITTI yields better performance even without intensity information.
• Systematic in-game scene sampling reveals near-field and off-center regions as weak spots (low IoU), enabling targeted retraining.
• Retraining with selected synthetic point clouds from weak regions substantially improves IoU across most positions while preserving KITTI performance.
• Using the configurable in-game scenes to test and retrain can fix weaknesses in the neural network without harming real-data accuracy.