[Paper Review] Crowdsourced 3D Mapping: A Combined Multi-View Geometry and Self-Supervised Learning Approach

Hemang Chawla, Matti Jukola+3 more|arXiv (Cornell University)|Jul 25, 2020

Robotics and Sensor-Based Localization48 references7 citations

TL;DR

This paper proposes a crowdsourced 3D traffic sign mapping framework that jointly leverages multi-view geometry and self-supervised deep learning to estimate camera intrinsics, depth, and ego-motion from monocular RGB video and GPS data—enabling accurate 3D sign positioning without prior knowledge of camera parameters. The method achieves a mean relative positioning error of 0.39 m and absolute error of 1.26 m across sequences, outperforming prior approaches that require known intrinsics or additional sensors.

ABSTRACT

The ability to efficiently utilize crowdsourced visual data carries immense potential for the domains of large scale dynamic mapping and autonomous driving. However, state-of-the-art methods for crowdsourced 3D mapping assume prior knowledge of camera intrinsics. In this work, we propose a framework that estimates the 3D positions of semantically meaningful landmarks such as traffic signs without assuming known camera intrinsics, using only monocular color camera and GPS. We utilize multi-view geometry as well as deep learning based self-calibration, depth, and ego-motion estimation for traffic sign positioning, and show that combining their strengths is important for increasing the map coverage. To facilitate research on this task, we construct and make available a KITTI based 3D traffic sign ground truth positioning dataset. Using our proposed framework, we achieve an average single-journey relative and absolute positioning accuracy of 39cm and 1.26m respectively, on this dataset.

Motivation & Objective

• To enable large-scale, cost-effective 3D traffic sign mapping using consumer-grade sensors without prior camera calibration.
• To address the challenge of unknown or varying camera intrinsics in crowdsourced visual data for 3D reconstruction.
• To improve map coverage and accuracy by fusing multi-view geometry and deep learning-based self-calibration, depth, and ego-motion estimation.
• To provide a publicly available KITTI-based 3D traffic sign ground truth dataset for benchmarking.

Proposed method

• The framework uses monocular RGB images and GPS data as input, eliminating reliance on calibrated sensors or IMUs.
• It applies multi-view geometry techniques (e.g., COLMAP) for self-calibration and ego-motion estimation via structure-from-motion and bundle adjustment.
• Self-supervised deep learning models (Monodepth2 and VITW) are used for monocular depth and ego-motion estimation, trained without ground-truth depth.
• A hybrid approach combines geometry-based tracking (ORB-SLAM with/without loop closure) and learned depth maps to triangulate 3D positions of traffic signs.
• Sign positions are estimated via triangulation using multiple views, with self-calibration results used to refine camera projection matrices.
• The system evaluates performance across sequences using relative and absolute positioning errors, with ablation studies on calibration and estimation methods.

Experimental results

Research questions

• RQ1 How does the accuracy of 3D traffic sign triangulation depend on the precision of self-calibration when camera intrinsics are unknown?
• RQ2 What is the comparative performance of geometry-based vs. deep learning-based methods for self-calibration, depth estimation, and ego-motion in crowdsourced mapping?
• RQ3 Can a hybrid framework combining multi-view geometry and self-supervised learning improve map coverage and 3D sign localization accuracy compared to pure geometry or learning-based approaches?
• RQ4 How does the use of GPS-only data with monocular vision compare to systems using additional sensors like IMUs or known intrinsics in terms of 3D sign positioning accuracy?

Key findings

• The proposed framework achieves a mean relative 3D traffic sign positioning error of 0.39 m and a mean absolute error of 1.26 m across all sequences, demonstrating high accuracy without known camera intrinsics.
• Combining ORB-SLAM with loop closure and COLMAP-based self-calibration yielded the best relative positioning performance, with a relative error of 0.24 m when normalized per sign.
• For sequences with tracking failure (e.g., Seq 1), Approach B using Monodepth2 for depth estimation and COLMAP for calibration achieved the best error (0.58 m relative error), showing robustness in challenging conditions.
• The use of self-calibration via COLMAP or VITW (m) significantly improved sign positioning accuracy compared to no calibration, confirming the importance of intrinsic estimation.
• The hybrid framework outperformed both pure geometry and pure deep learning approaches, especially in long trajectories and feature-poor environments.
• The open-sourced KITTI-based 3D traffic sign ground truth dataset enables reproducible evaluation and benchmarking of future crowdsourced mapping methods.