[Paper Review] MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception

Thien‐Minh Nguyen, Shenghai Yuan+9 more|arXiv (Cornell University)|Mar 18, 2024

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces MCD, a large-scale, multi-campus dataset featuring diverse sensing modalities—including low-cost NRE lidar, cameras, IMUs, and UWB—along with pointwise semantic annotations for 29 classes across 59k NRE scans. It proposes novel continuous-time ground truth via optimization-based lidar-inertial registration on survey-grade maps, revealing significant performance drops (>25% mIoU) in SOTA 3D semantic segmentation models when applied to NRE lidar, highlighting critical challenges in generalization and robustness.

ABSTRACT

Perception plays a crucial role in various robot applications. However, existing well-annotated datasets are biased towards autonomous driving scenarios, while unlabelled SLAM datasets are quickly over-fitted, and often lack environment and domain variations. To expand the frontier of these fields, we introduce a comprehensive dataset named MCD (Multi-Campus Dataset), featuring a wide range of sensing modalities, high-accuracy ground truth, and diverse challenging environments across three Eurasian university campuses. MCD comprises both CCS (Classical Cylindrical Spinning) and NRE (Non-Repetitive Epicyclic) lidars, high-quality IMUs (Inertial Measurement Units), cameras, and UWB (Ultra-WideBand) sensors. Furthermore, in a pioneering effort, we introduce semantic annotations of 29 classes over 59k sparse NRE lidar scans across three domains, thus providing a novel challenge to existing semantic segmentation research upon this largely unexplored lidar modality. Finally, we propose, for the first time to the best of our knowledge, continuous-time ground truth based on optimization-based registration of lidar-inertial data on large survey-grade prior maps, which are also publicly released, each several times the size of existing ones. We conduct a rigorous evaluation of numerous state-of-the-art algorithms on MCD, report their performance, and highlight the challenges awaiting solutions from the research community.

Motivation & Objective

• To address the lack of diverse, large-scale, and well-annotated datasets for robot perception beyond autonomous driving scenarios.
• To enable research on low-cost, non-repetitive epicyclic (NRE) lidar, a modality underexplored in semantic segmentation due to sparsity and irregular scanning.
• To provide high-accuracy, continuous-time ground truth for SLAM and localization by optimizing lidar-inertial data against survey-grade prior maps.
• To foster robust perception algorithms by exposing challenges such as lighting variations, glass reflections, and non-line-of-sight UWB effects.
• To support domain generalization research by collecting data across three geographically and environmentally diverse university campuses.

Proposed method

• The dataset was collected across three university campuses in Asia, Europe, and Germany, covering diverse environmental conditions and architectural styles.
• It integrates multiple sensors: classical rotating lidar (CCS), MEMS-based NRE lidar, high-frequency IMUs, RGB cameras with multiple baselines, and UWB anchors.
• Semantic annotations were manually created for 59,000 NRE lidar scans across 29 object classes, making it the first large-scale semantic dataset for this modality.
• Continuous-time ground truth was generated by optimizing lidar-inertial trajectories against pre-built, high-accuracy survey-grade maps using a nonlinear optimization framework.
• The ground truth supports arbitrary time and density sampling, enabling high-fidelity evaluation for AR/VR and real-time robotics applications.
• The dataset includes motion-distorted and motion-corrected lidar data, allowing researchers to study the impact of motion blur on perception models.

Experimental results

Research questions

• RQ1 Can state-of-the-art 3D semantic segmentation models trained on cylindrical lidar generalize to the sparser, irregular point clouds of NRE lidar?
• RQ2 How does domain shift across different campuses affect the performance of existing SLAM and perception algorithms?
• RQ3 To what extent does motion distortion in raw NRE lidar scans degrade the performance of perception systems, and how can it be mitigated?
• RQ4 Can continuous-time ground truth improve the accuracy and reliability of SLAM and localization in large-scale environments?
• RQ5 How do environmental challenges such as solar interference, glass reflections, and extreme lighting affect perception system robustness?

Key findings

• All SOTA 3D semantic segmentation models experienced a relative mIoU drop of over 25% when tested on NRE lidar scans compared to training on SemanticKITTI, indicating poor generalization to this modality.
• Methods relying on cylindrical or projection-based representations (e.g., SalsaNext, Cylinder3D) suffered the most severe performance degradation due to incompatibility with NRE point cloud patterns.
• Voxel-based models (e.g., MKNet, SPVCNN) also showed significant performance drops, primarily due to point density discrepancies between CCS and NRE lidars.
• Only WaffleIron and S-Former achieved mIoU scores above 50% on NRE scans, due to their flexible, point-wise representation learning that adapts to irregular scanning.
• When tested on a different campus (TUHH), all models saw a relative performance drop of over 70% compared to their performance on the NTU campus, indicating strong domain shift and poor cross-environment generalization.
• The study reveals a critical gap in current perception models' ability to handle real-world challenges such as solar interference, glass reflections, and non-line-of-sight UWB, which are underrepresented in existing benchmarks.