[Paper Review] VDO-SLAM: A Visual Dynamic Object-aware SLAM System

Jun Zhang, Mina Henein+2 more|arXiv (Cornell University)|May 22, 2020

Robotics and Sensor-Based Localization57 references110 citations

TL;DR

VDO-SLAM presents a robust, feature-based SLAM system that jointly localises the camera, tracks dynamic rigid objects, and builds a spatiotemporal map by leveraging semantic information and object-motion modeling in SE(3). It extracts object velocities and demonstrates improvements over state-of-the-art methods in real-world datasets.

ABSTRACT

Combining Simultaneous Localisation and Mapping (SLAM) estimation and dynamic scene modelling can highly benefit robot autonomy in dynamic environments. Robot path planning and obstacle avoidance tasks rely on accurate estimations of the motion of dynamic objects in the scene. This paper presents VDO-SLAM, a robust visual dynamic object-aware SLAM system that exploits semantic information to enable accurate motion estimation and tracking of dynamic rigid objects in the scene without any prior knowledge of the objects' shape or geometric models. The proposed approach identifies and tracks the dynamic objects and the static structure in the environment and integrates this information into a unified SLAM framework. This results in highly accurate estimates of the robot's trajectory and the full SE(3) motion of the objects as well as a spatiotemporal map of the environment. The system is able to extract linear velocity estimates from objects' SE(3) motion providing an important functionality for navigation in complex dynamic environments. We demonstrate the performance of the proposed system on a number of real indoor and outdoor datasets and the results show consistent and substantial improvements over the state-of-the-art algorithms. An open-source version of the source code is available.

Motivation & Objective

• Motivate robust navigation and mapping in dynamic environments where objects in motion affect SLAM accuracy.
• Develop a unified estimation framework that models camera poses, static 3D points, dynamic object motions, and object-level velocities without requiring prior object models.
• Leverage semantic information and dense feature representations to improve data association and tracking under occlusions.
• Demonstrate full dynamic SLAM performance on real indoor/outdoor datasets and provide open-source code.

Proposed method

• Formulate dynamic SLAM as a joint estimation problem over camera poses, static 3D points, and object motions in SE(3).
• Represent rigid object motion using a model-free SE(3) pose change that relates point tracks across frames.
• Estimate camera pose and object motion by minimizing 3D-2D re-projection errors on the image plane with Lie-algebra parametrization.
• Jointly refine optical flow, camera pose, and object motion to robustly track points under occlusions.
• Construct a factor-graph optimization (batch or incremental) incorporating 3D point measurements, odometry, object motion factors, and smooth-motion priors.
• Maintain a global map that includes static structure and dynamic object trajectories, enabling velocity extraction for objects.

Experimental results

Research questions

• RQ1 How can dynamic scenes be modeled within a unified SLAM framework without relying on predefined object models?
• RQ2 Can SE(3) object motions be robustly estimated from point tracks on moving rigid bodies using model-free representations?
• RQ3 Does integrating semantic instance segmentation with dense feature tracking improve data association and stability in dynamic SLAM?
• RQ4 How does joint optimization of camera pose, static points, and dynamic object motions compare to traditional static-SLAM or two-stage MOT approaches in real-world datasets?

Key findings

• The system achieves accurate SE(3) motion estimates for dynamic objects and recovers object velocities from scene motion.
• A robust object-tracking method leveraging semantic information can handle indirect occlusions from segmentation failures.
• VDO-SLAM demonstrates substantial improvements over state-of-the-art methods on real indoor and outdoor datasets.
• The approach provides a full dynamic SLAM system that estimates camera pose, static/dynamic structure, and the full 6-DoF object motions simultaneously.
• An open-source implementation of VDO-SLAM is available for researchers.