[Paper Review] CARLA-Loc: Synthetic SLAM Dataset with Full-stack Sensor Setup in Challenging Weather and Dynamic Environments

Yuhang Han, Zhengtao Liu+5 more|arXiv (Cornell University)|Sep 16, 2023

Robotics and Sensor-Based Localization4 citations

TL;DR

CARLA-Loc is a synthetic SLAM dataset generated in the CARLA simulator with full-stack sensor fusion (LiDAR, cameras, IMU, event camera) and precise calibration across 7 maps and 42 sequences under varying weather and dynamic conditions. It enables controlled, repeatable evaluation of SLAM algorithms, demonstrating that IMU integration and semantic segmentation-based dynamic object removal significantly improve localization accuracy in challenging environments.

ABSTRACT

The robustness of SLAM (Simultaneous Localization and Mapping) algorithms under challenging environmental conditions is critical for the success of autonomous driving. However, the real-world impact of such conditions remains largely unexplored due to the difficulty of altering environmental parameters in a controlled manner. To address this, we introduce CARLA-Loc, a synthetic dataset designed for challenging and dynamic environments, created using the CARLA simulator. Our dataset integrates a variety of sensors, including cameras, event cameras, LiDAR, radar, and IMU, etc. with tuned parameters and modifications to ensure the realism of the generated data. CARLA-Loc comprises 7 maps and 42 sequences, each varying in dynamics and weather conditions. Additionally, a pipeline script is provided that allows users to generate custom sequences conveniently. We evaluated 5 visual-based and 4 LiDAR-based SLAM algorithms across different sequences, analyzing how various challenging environmental factors influence localization accuracy. Our findings demonstrate the utility of the CARLA-Loc dataset in validating the efficacy of SLAM algorithms under diverse conditions.

Motivation & Objective

• To address the lack of controlled, repeatable benchmark datasets for evaluating SLAM robustness under dynamic and adverse weather conditions.
• To provide a synthetic dataset with consistent ego-motion trajectories across varying environmental disturbances for fair algorithm comparison.
• To enable quantitative analysis of how weather, dynamic objects, and sensor fusion impact SLAM performance using ground-truth annotations.
• To support research on dynamic-aware LiDAR SLAM by providing semantic segmentation labels for point cloud filtering.
• To serve as a standardized benchmark for evaluating visual, LiDAR, and multi-sensor SLAM algorithms under realistic but controlled challenges.

Proposed method

• Generated 42 sequences across 7 diverse maps (urban, rural, highway) using the CARLA simulator with identical ego-trajectories for lateral comparison.
• Integrated full-stack sensors: stereo cameras, depth camera, event camera, LiDAR, and IMU, with strict calibration, synchronization, and timestamping.
• Applied ground-truth semantic segmentation to both stereo images and LiDAR point clouds, enabling dynamic object point removal for evaluation.
• Enabled consistent environmental variation by controlling weather (clear, foggy, rainy) and dynamic object levels (static, dynamic) while preserving ego-motion.
• Used the CARLA recorder function to ensure identical trajectories across different weather and dynamic scenarios for fair performance comparison.
• Evaluated 5 visual-based and 4 LiDAR-based SLAM algorithms across sequences, comparing raw, segmented, and IMU-augmented results.

Experimental results

Research questions

• RQ1 How do varying weather conditions (e.g., fog, rain) affect the localization accuracy of visual and LiDAR-based SLAM algorithms?
• RQ2 To what extent do dynamic objects (e.g., vehicles, pedestrians) degrade SLAM performance, and how can this be mitigated?
• RQ3 How does the integration of IMU data improve the robustness of SLAM systems in dynamic and adverse weather conditions?
• RQ4 What is the impact of pre-processing LiDAR point clouds by removing points from dynamic objects using semantic segmentation on localization accuracy?
• RQ5 How do direct LiDAR odometry methods (e.g., FAST-LIO) compare to feature-based methods in terms of robustness to dynamic and adverse conditions?

Key findings

• The presence of dynamic objects significantly increases relative pose error in SLAM systems, with A-LOAM and LeGO-LOAM showing up to 147.27 mm error in map 07 under dynamic conditions.
• IMU integration substantially reduces localization error: FAST-LIO(F) achieved 1.95 mm relative pose error in dynamic sequences, compared to 16.72 mm without IMU in map 07.
• Semantic segmentation-based dynamic point removal reduced error in A-LOAM from 16.72 mm (raw dynamic) to 5.97 mm (segmented dynamic), showing measurable improvement.
• The direct LiDAR odometry method FAST-LIO2 (without feature extraction) demonstrated superior robustness to dynamic disturbances, outperforming feature-based variants in most sequences.
• LeGO-LOAM was most sensitive to dynamic objects due to its reliance on ground-plane features, showing the highest error increase (up to 147.27 mm) in dynamic scenes.
• In adverse weather, visual SLAM performance degraded significantly, with higher relative pose errors observed in rainy and foggy conditions, especially in sequences with high dynamic content.