[Paper Review] A Photometrically Calibrated Benchmark For Monocular Visual Odometry

Jakob Engel, Vladyslav Usenko+1 more|arXiv (Cornell University)|Jul 9, 2016

Robotics and Sensor-Based Localization17 references150 citations

TL;DR

Introduces the TUM monoVO benchmark with 50 real-world monocular VO/SLAM sequences that are photometrically calibrated, enabling evaluation via loop-closure drift without full ground-truth trajectories.

ABSTRACT

We present a dataset for evaluating the tracking accuracy of monocular visual odometry and SLAM methods. It contains 50 real-world sequences comprising more than 100 minutes of video, recorded across dozens of different environments -- ranging from narrow indoor corridors to wide outdoor scenes. All sequences contain mostly exploring camera motion, starting and ending at the same position. This allows to evaluate tracking accuracy via the accumulated drift from start to end, without requiring ground truth for the full sequence. In contrast to existing datasets, all sequences are photometrically calibrated. We provide exposure times for each frame as reported by the sensor, the camera response function, and dense lens attenuation factors. We also propose a novel, simple approach to non-parametric vignette calibration, which requires minimal set-up and is easy to reproduce. Finally, we thoroughly evaluate two existing methods (ORB-SLAM and DSO) on the dataset, including an analysis of the effect of image resolution, camera field of view, and the camera motion direction.

Motivation & Objective

• Provide a large-scale monocular VO/SLAM benchmark covering diverse environments to enable robust tracking accuracy evaluation without full ground-truth poses.
• Incorporate photometric calibration (response function and vignetting) and exposure times to reflect real sensor pipelines.
• Propose a loop-closure-based evaluation methodology that measures accumulated drift over long sequences.
• Offer simple, reproducible calibration procedures for non-parametric vignette calibration and photometric response.
• Publish code, raw data, and evaluation tools to encourage standardized benchmarking.

Proposed method

• Capture 50 sequences totaling 105 minutes with two fisheye lenses and varying frame rates.
• Calibrate geometry with a FOV distortion model suitable for fisheye optics.
• Estimate photometric calibration including camera response function G and vignette map V from multi-exposure static scenes.
• Use a non-parametric approach to vignette calibration requiring minimal setup.
• Define a loop-closure based evaluation that aligns start and end segments to measure drift, independent of full ground-truth trajectories.
• Evaluate ORB-SLAM and Direct Sparse Odometry (DSO) on the dataset under varying resolutions, FOVs, and motion patterns.

Experimental results

Research questions

• RQ1 How does photometric calibration (response function and vignetting) affect monocular VO/SLAM performance on real-world sequences?
• RQ2 What is the impact of image resolution, field of view, and camera motion direction on the accuracy and robustness of VO/SLAM methods on a large, diverse dataset?
• RQ3 Can loop-closure based drift metrics reliably compare monocular VO/SLAM methods without full ground-truth data?
• RQ4 How do state-of-the-art methods like ORB-SLAM and DSO perform across a varied set of indoor and outdoor environments when evaluated with photometric calibration?
• RQ5 What is the benefit of using a non-parametric vignette calibration approach in practical setups?

Key findings

• The dataset contains 50 sequences with 105 minutes of video and exposure-time metadata, enabling drift-based evaluation without full ground-truth trajectories.
• Photometric calibration includes camera response and dense vignette maps, improving evaluation realism for direct methods.
• Two baseline methods (ORB-SLAM and DSO) were evaluated, revealing sensitivity to field of view, resolution, and motion direction, with quantitative comparisons across multiple runs.
• The evaluation framework shows how drift location within a sequence affects error metrics, advocating the alignment-based “alignment error” as a robust overall measure.
• Results demonstrate that smaller FOVs degrade accuracy for both methods, while direct methods show different sensitivity to resolution (DSO less affected).
• The paper provides open-source code, raw data, and scripts for reproducing photometric calibration and evaluation.