[Paper Review] TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories

Jannik Zürn, S. M. Weber+1 more|arXiv (Cornell University)|Sep 12, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

TrackletMapper proposes a self-supervised framework for semantic segmentation of ground surfaces—such as sidewalks, roads, and crossings—using trajectories of pedestrians and vehicles, without human-annotated data. By projecting these tracklets into ego-view images and leveraging self-distillation via aggregated surface maps, the method achieves improved segmentation accuracy, especially in pedestrian-centric urban environments where domain shift from vehicle-centric datasets is a major challenge.

ABSTRACT

Robustly classifying ground infrastructure such as roads and street crossings is an essential task for mobile robots operating alongside pedestrians. While many semantic segmentation datasets are available for autonomous vehicles, models trained on such datasets exhibit a large domain gap when deployed on robots operating in pedestrian spaces. Manually annotating images recorded from pedestrian viewpoints is both expensive and time-consuming. To overcome this challenge, we propose TrackletMapper, a framework for annotating ground surface types such as sidewalks, roads, and street crossings from object tracklets without requiring human-annotated data. To this end, we project the robot ego-trajectory and the paths of other traffic participants into the ego-view camera images, creating sparse semantic annotations for multiple types of ground surfaces from which a ground segmentation model can be trained. We further show that the model can be self-distilled for additional performance benefits by aggregating a ground surface map and projecting it into the camera images, creating a denser set of training annotations compared to the sparse tracklet annotations. We qualitatively and quantitatively attest our findings on a novel large-scale dataset for mobile robots operating in pedestrian areas. Code and dataset will be made available at http://trackletmapper.cs.uni-freiburg.de.

Motivation & Objective

• To address the domain gap in semantic segmentation models trained on vehicle-centric datasets when deployed in pedestrian environments.
• To eliminate the need for expensive human annotation of pedestrian-view imagery by leveraging observable traffic participant trajectories.
• To develop a scalable, automatic annotation pipeline for multi-class ground surface segmentation (roads, sidewalks, crossings) from robot-collected data.
• To improve model generalization and performance through self-distillation using aggregated semantic surface maps.

Proposed method

• Project robot ego-trajectories and tracked object trajectories (pedestrians, vehicles) into ego-view camera images to generate sparse, multi-class pixel-level annotations.
• Use object type (e.g., pedestrian vs. vehicle) to infer ground surface labels: pedestrians imply sidewalks/pedestrian zones, vehicles imply roads or crossings.
• Construct a dense semantic surface map by spatially aggregating model predictions across frames to improve annotation consistency and coverage.
• Apply knowledge distillation by re-projecting the aggregated surface map back into the camera image space to generate synthetic, dense training labels.
• Train a semantic segmentation model end-to-end using the sparse tracklet annotations and the distilled, dense annotations for improved performance.
• Leverage a novel large-scale dataset, Freiburg Pedestrian Scenes, collected from a mobile robot navigating urban pedestrian areas.

Experimental results

Research questions

• RQ1 Can traffic participant trajectories be reliably used to generate semantic annotations for ground surfaces without human-annotated data?
• RQ2 How does the integration of tracklet-based annotations compare to ego-trajectory-only supervision in terms of segmentation accuracy across multiple ground surface classes?
• RQ3 To what extent can self-distillation via aggregated surface maps improve model performance compared to standard supervision?
• RQ4 How robust is the annotation pipeline to real-world variations such as inconsistent traffic behavior or sensor noise?
• RQ5 Can the proposed method generalize to complex urban environments with visually similar ground surfaces like sidewalks and crossings?

Key findings

• The tracklet-based annotation method outperforms ego-trajectory-only supervision across all classes except obstacles, demonstrating the value of leveraging diverse traffic participant behaviors.
• Self-distillation using aggregated surface maps leads to a measurable improvement in IoU scores, attributed to increased annotation density and consistency.
• The model trained on TrackletMapper annotations generalizes better than models trained on Vistas, reducing viewpoint-induced biases common in vehicle-centric datasets.
• Qualitative results show that segmentation masks are well-aligned with ground truth, though challenges remain in distinguishing crossings from adjacent road and sidewalk regions.
• Limitations include class bleeding in overlapping regions (especially Crossing), and sensitivity to inaccurate localization, calibration, or rule-breaking traffic behavior such as jaywalking.